Skip to content

2. Azure Synchronization

Jump to bottom
Danny Collier edited this page Apr 12, 2025 · 8 revisions

THESE REQUIREMENTS ARE IN A VERY EARLY DRAFT

Overview

The ultimate deliverables for our application will be a set of xml, txt, csv, and media files delivered to Azure file storage. (Note that this is Azure file storage and not an Azure storage blob.) SkillRX will store and manage its media files in its own file storage. A process within SkillRX will manage synchronization with Azure.

The Azure storage includes folders for delivery to:

  • Raspberry devices (CMES-PI)
  • Mini computers
  • USB

As of this writing, we are not delivering files for USB. That could change.

Storage Paths and Filenames

Documents are stored in a folder in the Azure file storage based on their topic's language. The path comes down to:

  • "[environment-specific route]/[file share folder determined by language]/assets/[topic_id]_[filename_with_extension]"

For now, these requirements are only concerned with the "for CMES-PI" files, both current and legacy, and not the legacy USB app files.

We will generate and deliver one set of files for each language, delivered to the root storage path for the environment plus the paths and filenames specified here:

  • Legacy XML for CMES-Pi: [language.file_storage_prefix]CMES-Pi/assets/XML/[language.file_storage_prefix]Server_XML.xml
  • Provider XML for CMES-Pi: [language.file_storage_prefix]CMES-Pi/assets/XML/[language.file_storage_prefix][provider name].xml for every provider
  • New topics for CMES-Pi: [language.file_storage_prefix]CMES-Pi/assets/XML/[language.file_storage_prefix]New_Uploads.xml
  • We will not generate Top topics for CMES-Pi: [language.file_storage_prefix]CMES-Pi/assets/XML/[language.file_storage_prefix]Top_Topics.xml as these depend on collecting stats from remote devices.

These file names and locations follow the patterns described in the XML Generation application.

From the XML File Generation app documentation: image

Filenames will have topic.id and an underscore prepended to prevent filename collisions. It is less likely but still possible for an editor to upload two files with the same name to the same topic.

When a topic is archived, we will move its related Training Materials documents to an archival folder. Archival folders follow this naming convention:

  • [language.file_storage_prefix]CMES-Pi_Archive/[their unarchived location following the conventions described above.

Media Files

Media storage follows this convention:

  • [language.file_storage_prefix]CMES-Pi/assets/Content/[Topic_id]_[file name with underscores instead of spaces and no special characters].[file extension]

XML Files

See the XML Generation application for details.

"Provider XML" structure from the XML Generation app documentation: image

"Legacy XML" structure from the XML Generation app documentation: image

Tag Files

In addition to the XML and the media files, we will deliver .txt files containing tag information.

We will generate and deliver one set of files for each language, delivered to the root storage path for the environment plus the paths and filenames specified here:

  • Tag file for CMES-Pi: [language.file_storage_prefix]CMES-Pi/assets/Tags/[language.file_storage_prefix]tags.txt
  • Tags and title file for CMES-Pi: [language.file_storage_prefix]CMES-Pi/assets/Tags/[language.file_storage_prefix]tagsAndTitle.txt

CSV Files

The mini computers rely on a set of .csv files which they import to their local database. [full requirements for those files]

Azure Client

We will write a client which we can use to log into Azure and manage files using the Azure REST API for file storage. We may not need all of the available REST actions for our purposes.

We have not found a viable way to set up Azure file storage for local development environments, so we will use a shared Azure file storage created by our stakeholders. We will provide the name of this environment and access details to the developers working on the Azure synchronization. Since this is a shared environment, developers will need to coordinate: their local development environments will all be writing to the same storage. (If necessary, we should consider creating individual developer folders inside that storage.)

The Azure client will be used to:

  • Authenticate
  • Add individual media, csv, txt, and xml files to Azure.
  • Archive files when a topic is archived. There is no api for moving files, so we will delete the file and upload it to the archive location (see above)
  • Delete files when a topic is deleted

Synchronization

[Requirements not yet written. Are we doing this synchronously or asynchronously for topic media files? How often are we updating the xml, csv, and txt files? How can we optimize this so we do not update anything unnecessarily? There should be no synchronization resulting from the import. Etc.]

Clone this wiki locally