feat: implement submission file workflow with comprehensive testing

- Create Submission file model
- Develop SubmissionFileManager for centralized DB query handling
- Add TestSubmissionFileManager with extensive test coverage
  - Test file processing with different input types (bytes, file objects)
  - Implement error handling tests for IO errors and invalid files
  - Verify complete file processing workflow
  - Validation tests for file addition in create_external_grader_detail
- Document architectural decisions with ADR
- Add test queue folder in .gitignore

Author: leoaulasneo98

.gitignore

@@ -63,3 +63,4 @@ test.txt.tmp
 
 # VSCode
 .vscode
+test_queue/*

docs/source/decisions/0002-add-submission-file-model.rst

@@ -0,0 +1,150 @@
+2. File Handling Implementation for Submission System
+#####################################################
+
+Status
+******
+
+**Provisional** *2025-02-14*
+
+Implemented by: waiting for it
+
+Context
+*******
+
+As part of the XQueue migration effort detailed in ADR 1, we need to handle file submissions that were previously
+managed by the XQueue system. The existing implementation lacks dedicated file handling capabilities within
+edx-submissions, requiring:
+
+1. File Management:
+   - Secure storage of student-submitted files
+   - Association with specific submissions
+   - Compatibility with XQueue URL formats
+   - Support for multiple file types and formats
+
+2. Processing Requirements:
+   - Robust error handling for file operations
+   - Validation of file objects
+   - Dynamic path generation
+   - Proper cleanup and resource management
+
+3. Integration Points:
+   - Connection with SubmissionQueueRecord
+   - Support for existing xqueue-watcher interface
+   - File URL format compatibility
+   - Efficient database querying
+
+The current XQueue system implements file handling through its Submission model, which manages file storage and
+retrieval in a tightly coupled way. The existing implementation:
+
+1. Current File Management in XQueue:
+   - Uses a Submission model with direct file storage fields (s3_keys and s3_urls)
+   - Handles file uploads through manual storage management functions (upload_file_dict, upload)
+   - Stores file URLs as JSON strings in model fields
+   - Requires synchronous HTTP communication for file retrieval
+
+2. Submission Model File Fields:
+   ```python
+   s3_keys = models.CharField(max_length=CHARFIELD_LEN_LARGE)  # keys for internal use
+   s3_urls = models.CharField(max_length=CHARFIELD_LEN_LARGE)  # urls for external access
+   ```
+
+3. Current Workflow Issues:
+   - File handling is tightly coupled with submission processing
+   - Relies on manual URL construction and string manipulation
+   - Lacks proper file validation and type checking
+   - Limited by CharField size for storing file information
+   - Requires complex get_submission logic for file retrieval
+
+4. Xqueue-watcher Integration:
+   - External graders rely on specific URL formats
+   - File access depends on HTTP-based retrieval
+   - File information is embedded in submission payload
+
+Decision
+********
+
+We will implement a new SubmissionFile model and supporting infrastructure to handle file management within
+edx-submissions:
+
+1. Core Model Structure:
+   - SubmissionFile model with UUID-based identification
+   - Foreign key relationship to SubmissionQueueRecord
+   - Django FileField for actual file storage
+   - Original filename preservation
+   - Creation timestamp tracking
+   - Composite index for efficient querying
+
+2. File Management Layer:
+   - SubmissionFileManager class for encapsulated file operations
+   - Robust file processing with multiple format support
+   - Error handling for various file-related exceptions
+   - XQueue-compatible URL generation
+
+3. Integration with Submission Creation:
+   - Extended create_external_grader_detail function
+   - File processing during submission creation
+   - Automatic file manager instantiation
+   - Error handling and logging
+
+Consequences
+************
+
+Positive:
+---------
+
+1. Architecture:
+   - Clean separation of concerns for file handling
+   - Improved maintainability through dedicated models
+   - Better error handling and logging
+   - Efficient database querying through proper indexing
+
+2. Integration:
+   - Seamless xqueue-watcher compatibility
+   - Support for existing file processing workflows
+   - Minimal changes required to client code
+
+3. Operations:
+   - More robust file processing
+   - Better tracking of file submissions
+   - Improved error visibility
+   - Simplified file management
+
+Negative:
+---------
+
+1. Technical Impact:
+   - Additional database tables and indexes
+   - Increased storage requirements
+   - More complex submission creation flow
+
+2. Migration Considerations:
+   - Temporary increased system complexity
+   - Additional testing requirements
+
+3. Performance:
+   - Additional database operations during submission
+   - File processing overhead
+
+References
+**********
+
+Current System Documentation:
+   * XQueue Repository: https://github.com/openedx/xqueue
+   * XQueue Watcher Repository: https://github.com/openedx/xqueue-watcher
+
+Related Repositories:
+   * edx-submissions: https://github.com/openedx/edx-submissions
+   * edx-platform: https://github.com/openedx/edx-platform
+   * XQueue Repository: https://github.com/openedx/xqueue
+   * edx-submissions: https://github.com/openedx/edx-submissions
+
+Related Documentation:
+   * ADR 1: Creation of SubmissionQueueRecord Model for XQueue Migration
+   * File Processing Documentation: https://github.com/openedx/edx-submissions/tree/master/docs
+
+Future Event Integration:
+   * Open edX Events Framework: https://github.com/openedx/openedx-events
+   * Event Bus Documentation: https://openedx.atlassian.net/wiki/spaces/AC/pages/124125264/Event+Bus
+
+Related Architecture Documents:
+   * Open edX Architecture Guidelines: https://openedx.atlassian.net/wiki/spaces/AC/pages/124125264/Architecture+Guidelines

submissions/api.py

@@ -28,6 +28,7 @@
     ScoreSummary,
     StudentItem,
     Submission,
+    SubmissionFileManager,
     score_reset,
     score_set
 )
@@ -96,6 +97,12 @@ def create_external_grader_detail(student_item_dict,
             points_possible=points_possible,
 
         )
+
+        files_dict = external_grader_additional_data.get('files')
+        if files_dict:
+            file_manager = SubmissionFileManager(instance)
+            file_manager.process_files(files_dict)
+
         return instance
 
     except DatabaseError as error:

submissions/migrations/0005_submissionfile.py

@@ -0,0 +1,32 @@
+# Generated by Django 4.2.17 on 2025-03-14 22:35
+
+from django.db import migrations, models
+import django.db.models.deletion
+import django.utils.timezone
+import submissions.models
+import uuid
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('submissions', '0004_externalgraderdetail'),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name='SubmissionFile',
+            fields=[
+                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('uid', models.UUIDField(default=uuid.uuid4, editable=False)),
+                ('file', models.FileField(max_length=512, upload_to=submissions.models.submission_file_path)),
+                ('original_filename', models.CharField(max_length=255)),
+                ('created_at', models.DateTimeField(default=django.utils.timezone.now)),
+                ('submission_queue', models.ForeignKey(null=True, on_delete=django.db.models.deletion.SET_NULL,
+                                                       related_name='files', to='submissions.externalgraderdetail')),
+            ],
+            options={
+                'indexes': [models.Index(fields=['submission_queue', 'uid'], name='submissions_submiss_b1d4b0_idx')],
+            },
+        ),
+    ]

submissions/models.py

@@ -10,11 +10,14 @@
 """
 
 import logging
+import os
 from datetime import timedelta
 from uuid import uuid4
 
 from django.conf import settings
 from django.contrib import auth
+from django.core.files.base import ContentFile
+from django.core.files.uploadedfile import SimpleUploadedFile
 from django.db import DatabaseError, models, transaction
 from django.db.models.signals import post_save, pre_save
 from django.dispatch import Signal, receiver
@@ -693,3 +696,102 @@ def update_status(self, new_status):
     def create_from_uuid(cls, submission_uuid, **kwargs):
         submission = Submission.objects.get(uuid=submission_uuid)
         return cls.objects.create(submission=submission, **kwargs)
+
+
+def submission_file_path(instance, _):
+    """
+    Generate file path for submission files.
+    Format: queue_name/uuid
+    The filename is replaced with the UUID to ensure uniqueness without preserving extension.
+    """
+    return os.path.join(
+        instance.submission_queue.queue_name,
+        f"{instance.uid}"
+    )
+
+
+class SubmissionFile(models.Model):
+    """
+    Model to handle files associated with submissions
+    """
+    uid = models.UUIDField(default=uuid4, editable=False)
+    # legacy S3 key
+    submission_queue = models.ForeignKey(
+        'submissions.ExternalGraderDetail',
+        on_delete=models.SET_NULL,
+        related_name='files',
+        null=True,
+    )
+    file = models.FileField(
+        upload_to=submission_file_path,
+        max_length=512
+    )
+    original_filename = models.CharField(max_length=255)  # This is necessary to send file name to xqueue-watcher
+    created_at = models.DateTimeField(default=now)
+
+    class Meta:
+        indexes = [
+            models.Index(fields=['submission_queue', 'uid']),
+        ]
+
+    @property
+    def xqueue_url(self):
+        """
+        Returns URL in xqueue format: /queue_name/uid
+        """
+        return f"/{self.submission_queue.queue_name}/{self.uid}"
+
+
+class SubmissionFileManager:
+    """
+    Manages file operations for submissions
+    """
+
+    def __init__(self, submission_queue):
+        self.submission_queue = submission_queue
+
+    def process_files(self, files_dict):
+        """
+        Process uploaded files.
+        Returns URLs in xqueue compatible format.
+        """
+        files_urls = {}
+        for filename, file_obj in files_dict.items():
+            if not (isinstance(file_obj, (bytes, ContentFile, SimpleUploadedFile)) or hasattr(file_obj, 'read')):
+                logger.warning(f"Invalid file object type for {filename}")
+                continue
+
+            if hasattr(file_obj, 'read'):
+                try:
+                    file_content = file_obj.read()
+                    if isinstance(file_content, bytes):
+                        file_obj = ContentFile(file_content, name=filename)
+                    else:
+                        continue
+                except (IOError, OSError) as e:
+                    logger.error(f"Error reading file {filename}: {e}")
+                    continue
+                except UnicodeDecodeError as e:
+                    logger.error(f"Error decoding file {filename}: {e}")
+                    continue
+
+            if isinstance(file_obj, bytes):
+                file_obj = ContentFile(file_obj, name=filename)
+
+            submission_file = SubmissionFile.objects.create(
+                submission_queue=self.submission_queue,
+                file=file_obj,
+                original_filename=filename
+            )
+            files_urls[filename] = submission_file.xqueue_url
+
+        return files_urls
+
+    def get_files_for_grader(self):
+        """
+        Returns files in format expected by xwatcher
+        """
+        return {
+            file.original_filename: file.file.url
+            for file in self.submission_queue.files.all()
+        }

submissions/tests/test_api.py

@@ -10,6 +10,7 @@
 import pytz
 # Django imports
 from django.core.cache import cache
+from django.core.files.uploadedfile import SimpleUploadedFile
 from django.db import DatabaseError, IntegrityError, connection, transaction
 from django.test import TestCase
 from django.utils.timezone import now
@@ -923,3 +924,55 @@ def test_create_multiple_submission_external_grader_details(self):
         self.assertEqual(submission2.answer, ANSWER_TWO)
 
         self.assertNotEqual(external_grader_detail1.submission.uuid, external_grader_detail2.submission.uuid)
+
+    def test_create_external_grader_detail_with_files(self):
+        """Test creating a queue record with file handling."""
+
+        test_file = SimpleUploadedFile(
+            "test.txt",
+            b"test content",
+            content_type="text/plain"
+        )
+
+        event_data = {
+            'queue_name': 'test_queue',
+            'files': {'test.txt': test_file}
+        }
+
+        external_grader_detail = api.create_external_grader_detail(STUDENT_ITEM, ANSWER_ONE, **event_data)
+
+        self.assertEqual(external_grader_detail.queue_name, 'test_queue')
+        self.assertEqual(external_grader_detail.files.count(), 1)
+        submission_file = external_grader_detail.files.first()
+        self.assertEqual(submission_file.original_filename, 'test.txt')
+
+    def test_create_external_grader_detail_with_multiple_files(self):
+        """Test creating a queue record with multiple files."""
+        test_files = {
+            'test1.txt': SimpleUploadedFile(
+                "test1.txt",
+                b"test content 1",
+                content_type="text/plain"
+            ),
+            'test2.txt': SimpleUploadedFile(
+                "test2.txt",
+                b"test content 2",
+                content_type="text/plain"
+            )
+        }
+
+        external_grader_detail = {
+            'queue_name': 'test_queue',
+            'files': test_files
+        }
+        external_grader_detail = api.create_external_grader_detail(STUDENT_ITEM, ANSWER_ONE, **external_grader_detail)
+
+        self.assertEqual(external_grader_detail.files.count(), 2)
+        filenames = set(external_grader_detail.files.values_list('original_filename', flat=True))
+        self.assertEqual(filenames, {'test1.txt', 'test2.txt'})
+
+    def test_create_external_grader_detail_without_files(self):
+        """Test creating a queue record without any files still works."""
+        external_grader_instance = api.create_external_grader_detail(STUDENT_ITEM, ANSWER_ONE, queue_name="test_queue")
+        self.assertEqual(external_grader_instance.queue_name, 'test_queue')
+        self.assertEqual(external_grader_instance.files.count(), 0)