FC-73 Xqu add submission file (Feature: Submission File System - Enhanced File Management for Graded Submissions) #286

leoaulasneo98 · 2025-02-24T15:19:23Z

FC-73 Feature: Submission File System - Enhanced File Management for Graded Submissions

Description

This pull request implements a comprehensive file management system for submissions through the introduction of the SubmissionFile model and its associated SubmissionFileManager. This enhancement supports proper storage, retrieval, and processing of files attached to submissions within the Open edX ecosystem.

Motivation

The existing submission system requires enhanced file handling capabilities to:

Provide seamless transition for existing services (xwatcher)
Establish reliable file storage with appropriate metadata
Integrate seamlessly with the new submission queue architecture

Previously, the XQueue server managed files for submissions that required document attachments. However, this was implemented inefficiently using plain text fields (s3_keys and s3_urls) without proper structure or validation. This approach led to inconsistent file handling, potential security issues, and maintenance challenges as the system scaled.

Key Improvements

Model Enhancements

Introduced SubmissionFile model with:
- Secure file storage mechanisms
- Proper association with submission queue records
- Standardized URL generation for grader access
- Original filename preservation

File Processing System

Developed SubmissionFileManager with capabilities for:
- Processing multiple file formats (bytes, file objects)
- Robust error handling for file processing issues
- Consistent URL generation for xqueue compatibility
- Efficient retrieval methods for external grader integration

Error Handling

Implemented comprehensive error handling for:
- Invalid file objects
- IO and OS exceptions during file reading
- Unicode decoding errors
- Non-standard file formats

Technical Details

File Processing

Support for multiple input types including:
- Raw bytes
- Django ContentFile objects
- UploadedFile instances
- Custom file-like objects with read() methods

Integration Points

Seamless connection with ExternalGraderDetail for complete submission workflow
Compatible interface for existing xqueue consumers

Testing Strategy

Extensive test coverage includes:

Various file input types
Complete processing workflow verification
Edge case handling for invalid files
Error handling for exceptional conditions
File retrieval in grader-compatible format

Open edX Compliance

This implementation adheres to Open edX standards through:

Modular design with clear separation of concerns
Comprehensive test coverage of edge cases
Well-documented interfaces
Consistent error handling patterns

Performance Considerations

Efficient file storage with minimal overhead
Optimized database access patterns
Careful handling of file objects to prevent memory issues

BREAKING CHANGES: None. Designed for full compatibility with existing systems.

openedx-webhooks · 2025-02-24T15:19:29Z

Thanks for the pull request, @leoaulasneo98!

This repository is currently maintained by @openedx/committers-edx-submissions.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
- This process (including the steps you'll need to take) is documented here.
If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

Dependencies

This PR must be merged before / after / at the same time as ...
Blockers

This PR is waiting for OEP-1234 to be accepted.
Timeline information

This PR must be merged by XX date because ...
Partner information

This is for a course on edx.org.
Supporting documentation
Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

The size and impact of the changes that it introduces
The need for product review
Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

codecov · 2025-02-24T15:20:57Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.92%. Comparing base (2413e6a) to head (ee29b2e).

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #286      +/-   ##
==========================================
+ Coverage   94.43%   94.92%   +0.48%     
==========================================
  Files          18       19       +1     
  Lines        2263     2481     +218     
  Branches       95      101       +6     
==========================================
+ Hits         2137     2355     +218     
  Misses        115      115              
  Partials       11       11

Flag	Coverage Δ
unittests	`94.92% <100.00%> (+0.48%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

angonz · 2025-03-06T13:22:36Z

Review should only start after PR #283 is merged.

angonz · 2025-04-03T19:44:40Z

docs/source/decisions/0002-add-submission-file-model.rst

Update author of cange
Add link to first ADR
Summarize and simplify.

angonz · 2025-04-03T19:51:53Z

submissions/models.py

+        """
+        files_urls = {}
+        for filename, file_obj in files_dict.items():
+            if not (isinstance(file_obj, (bytes, ContentFile, SimpleUploadedFile)) or hasattr(file_obj, 'read')):


Why check for hasattr(file_obj, 'read')?

angonz · 2025-04-03T19:56:54Z

submissions/models.py

+                logger.warning(f"Invalid file object type for {filename}")
+                continue
+
+            if hasattr(file_obj, 'read'):


Please explain this workflow

- Create Submission file model - Develop SubmissionFileManager for centralized DB query handling - Add classes to handle file errors - Add TestSubmissionFileManager with extensive test coverage - Test file processing with different input types (bytes, file objects) - Implement error handling tests for IO errors and invalid files - Verify complete file processing workflow - Validation tests for file addition in create_external_grader_detail - Document architectural decisions with ADR - Add test queue folder in .gitignore

angonz

Ok on my side. Please open for Axim's review.

openedx-webhooks added the open-source-contribution PR author is not from Axim or 2U label Feb 24, 2025

mphilbrick211 added the FC Relates to an Axim Funded Contribution project label Feb 24, 2025

angonz mentioned this pull request Mar 6, 2025

FC-73 Add XqueueViewset with xwatcher services (Feature: XQueueViewSet - Compatible API Endpoints for External Grader Integration) #287

Draft

leoaulasneo98 force-pushed the XQU-add-submission-file branch 3 times, most recently from 9d2c25b to f7d55d7 Compare March 17, 2025 18:14

leoaulasneo98 force-pushed the XQU-add-submission-file branch 4 times, most recently from 8b556bf to 5f37f2e Compare March 26, 2025 14:10

leoaulasneo98 force-pushed the XQU-add-submission-file branch from 5f37f2e to b9ce70a Compare April 3, 2025 12:45

angonz reviewed Apr 3, 2025

View reviewed changes

leoaulasneo98 force-pushed the XQU-add-submission-file branch from b9ce70a to ee29b2e Compare April 4, 2025 17:43

angonz approved these changes Apr 7, 2025

View reviewed changes

leoaulasneo98 marked this pull request as ready for review April 7, 2025 14:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FC-73 Xqu add submission file (Feature: Submission File System - Enhanced File Management for Graded Submissions) #286

FC-73 Xqu add submission file (Feature: Submission File System - Enhanced File Management for Graded Submissions) #286

leoaulasneo98 commented Feb 24, 2025 •

edited

Loading

openedx-webhooks commented Feb 24, 2025 •

edited

Loading

codecov bot commented Feb 24, 2025 •

edited

Loading

angonz commented Mar 6, 2025

angonz Apr 3, 2025

angonz Apr 3, 2025

angonz Apr 3, 2025

angonz left a comment

FC-73 Xqu add submission file (Feature: Submission File System - Enhanced File Management for Graded Submissions) #286

Are you sure you want to change the base?

FC-73 Xqu add submission file (Feature: Submission File System - Enhanced File Management for Graded Submissions) #286

Conversation

leoaulasneo98 commented Feb 24, 2025 • edited Loading

FC-73 Feature: Submission File System - Enhanced File Management for Graded Submissions

Description

Motivation

Key Improvements

Model Enhancements

File Processing System

Error Handling

Technical Details

File Processing

Integration Points

Testing Strategy

Open edX Compliance

Performance Considerations

openedx-webhooks commented Feb 24, 2025 • edited Loading

codecov bot commented Feb 24, 2025 • edited Loading

Codecov Report

angonz commented Mar 6, 2025

angonz Apr 3, 2025

Choose a reason for hiding this comment

angonz Apr 3, 2025

Choose a reason for hiding this comment

angonz Apr 3, 2025

Choose a reason for hiding this comment

angonz left a comment

Choose a reason for hiding this comment

leoaulasneo98 commented Feb 24, 2025 •

edited

Loading

openedx-webhooks commented Feb 24, 2025 •

edited

Loading

codecov bot commented Feb 24, 2025 •

edited

Loading