Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
refactor: major fixes and restructuring
Changes: - Convert the program into a package with separate modules for better organisation - Support running directly with uvicorn via 'python -m uvicorn gallery_dl_server:app' and as a package via 'python -m gallery_dl_server' - Update README with important clarifications and instructions - Move important templates to new 'docs' subfolder - Update file paths in workflow - Update '.ignore' files - Add '.gitattributes' file for specifying line endings - New default configuration file with more options and better formatting - Terminate the startup shell script if it is not executed inside the Docker container - Print an empty line to separate the startup logs and the server logs - Remove redundant update function and update route - Do not create and take ownership of the '/.cache/pip' and '/.local' directories inside the Docker container as permissions issues have been resolved and it is no longer intended for project dependencies to be updated inside the container - Support loading from multiple configuration file locations: if loading is a success, log the list of config files loaded; if loading was a failure due to an OSError (exit code 1), log the list of valid config file paths - Only copy the default configuration file into the directory mounted to '/config' inside the Docker container if none of the multiple valid configuration files are present - Use the default OS-dependent gallery-dl configuration file locations if not running in the Docker container - Moved functions for manipulating the config dict to new 'config' module - Function for removing values from the config dict can now remove two consecutive items from nested lists based on the value of the second item; this is useful for removing gallery-dl command line arguments that accept a value where the argument and its value are separate list items - New function for adding entries recursively to the config dict without unintentionally overwriting existing values - Run the download script in a subprocess as a module (via Python's '-m' flag) to allow the subprocess to use relative imports and share its logging configuration with the main process - Load and update the gallery-dl configuration inside the download subprocess to allow the download job to access the loaded configuration - Initiate gallery-dl downloads in the subprocess via its 'job' module instead of invoking the subprocess with a system call directly to gallery-dl; this gives more fine-grained control and crucially prevents initialisation of the config dict which fixes the overriding options by preserving modifications to the config dict throughout the download subprocess - Fix 'extract-audio' override option - New functions located in the 'output' module to configure logging for the main process and gallery-dl - Configure gallery-dl logging manually via its own 'output' module and set up a separate logger for writing unsupported URLs to a designated 'unsupported file' - Configure logging for gallery-dl only after the configuration file is loaded to allow the config file options to be used for the logging configuration - Use a custom logger for the main process that does not propagate to the root logger, which is configured and utilised instead by gallery-dl - Exception handling for the gallery-dl download job which logs the name of the exception and returns the correct exit code - Ensure the subprocess stdout is flushed each time it is written to so that log messages arrive in the intended order - Determine the piped subprocess stdout log message level by prepending a custom prefix to the log message, which is removed before the message is logged - If running in the Docker container, save a backup copy of the log file if it is not empty on server shutdown to '/config/logs' and append the filename with the date and time so that previous container logs will not be overwritten - Force the file handler to open the log file with UTF-8 encoding for compatibility
- Loading branch information
