User Guide

This section is designed for anyone using the Archival module. It will guide you on how to efficiently store and access your scientific data with ease for archival purposes.

Navigation

The following actions can be performed in the Archival module, which can be accessed through the menu option.

• New Folder
• Folder Activities
• Apply Legal Hold
• Upload File
• Add to Favorites
• Follow

File/Folder List

If a folder is listed in the bucket but the files are not present, you can create the folders by clicking on New Folder on the Archival Explorer page.

To receive notifications or emails about a folder or file, click the Follow button. Navigating to a specific path will display the folder or file list under that path.

Metadata Tagging

Metadata tags provide you an additional information about files and folders in key-value pairs. Tagged metadata facilitates easy retrieval of files and folders. Providing accurate and clear metadata will enhance file and folder retrieval through Advanced Search.

Example:

Consider the project "XYZ", there is a metadata file. Within this file, you will find the metadata key "Sample ID" with the corresponding value "1." In this context, "Sample ID" is the metadata key, and "1" is its associated value.

There are four types of metadata in Archival are listed below with the details:

System Defined Metadata

These values are generated by the system and cannot be edited. They are automatically updated by the system whenever changes occur. The following are the system-defined metadata keys:

• Created By
• Created Time
• Updated By
• Updated Time
• Size
• File Source (This is applicable only for files, and it can be Manual, System, or External)
• Linked bucket name (Applicable only for Virtual Folder)
• Previous bucket name (Applicable only for Virtual Folder

File source metadata indicates the origin of the file upload. For instance, if a file is uploaded manually, the value for this metadata will be 'manual.'

Archival Policy Metadata

The Archival Policy section contains metadata related to archival conditions, outlining the phases of a file or folder's lifecycle in Archival. The following are the archival-policy metadata keys:

• End Date
• Archive Period
• Retention Period
• Restore Period

User Defined Metadata

Archival enables you to add additional metadata to files and folders using metadata tags by associating specific metadata keys with them. These metadata keys and their expected data types must be predefined in the Metadata Management system.

You can only assign metadata keys that are part of the platform's Metadata Management, allowing you to provide the corresponding values.

User-defined metadata can be edited as long as the file or folder remains in primary storage. Once it moves to secondary storage, the metadata values become non-editable. However, if necessary, you can restore the folder to primary storage to make its metadata editable again.

The example of Metadata is shown in the below table

Inherited Metadata

"Inherited" means that any folder or file created within a parent folder will inherit the metadata from the parent, allowing them to be searched using the same metadata. However, the metadata can only be edited at the parent level where it was originally assigned.

User-defined and archival policies are inherited metadata that is passed from a parent folder to its child files and folders. Since this metadata is inherited, it is non-editable for the child files and folders. However, you can edit the metadata in the parent folder or override the values in the child. Inherited metadata keys are marked with the special symbol that show in the below image.

Add Metadata

You can associate metadata keys with a folder during its creation. Use the Add Row field to add the necessary number of rows based on the number of metadata entries to be associated.

Associate the required metadata keys and provide corresponding values. Repeat this process to add additional metadata keys as needed.

You cannot associate the same user-defined metadata key more than once with a file or folder, even if the values are different. Use a different metadata key for additional entries. Metadata keys can also be associated on the Edit Properties page.

View Metadata

To view the metadata keys and values, use the 'Side Panel' icon on the file explorer page. The metadata associated with the folder can be found under the 'Metadata' tab, where you can view both user-defined and inherited metadata.

To view system-defined and archival policy metadata, switch to the 'Folder Details' tab.

When there are many metadata keys, you can narrow down your search by using the search and filter options. Here,

• Search for metadata keys using the Search bar
• Filter metadata based on their type: User Defined or Inherited

Each metadata key will have a specific datatype, which can be identified by its corresponding symbol. The following table provides the different metadata keys along with their symbols.

Remove Metadata

Once you have associated certain metadata keys, you have the option to dissociate a different set of metadata keys. Use the delete icon to remove the metadata key association from the folder.

You can also remove multiple metadata key associations by selecting them. Confirm the delete action to remove the selected metadata associations.

Metadata Inheritance

All user-defined metadata set for a parent folder will be inherited by its child folders and files. Unless metadata is explicitly overridden or re-entered in the child items, they will inherit the metadata values from the parent. This section explains how to override a parent’s metadata.

Ensure that the parent folder has associated metadata, which is inherited by its child folders. In the 'Inherited' section, you can view the metadata inherited by the folder.

You can override inherited metadata in a child folder by selecting the same metadata key and providing a different value. When the folder is created, the child's metadata will replace the parent's metadata for that key.

Archival Policy metadata will be inherited by both child folders and files. While you can override this policy for child folders, it cannot be overridden for child files.

📘

NOTE

You will be able to view both the inherited metadata keys and their values, as well as any overridden metadata keys and their values, for files.

Manual Upload

To upload files to a folder, you must either be an Archival User with permissions to view files, view metadata, manage metadata, view folder audits, and assign archival roles, or an Archival Super User with additional permissions to view folders, create folders, restore folders, place legal holds, upload files, and view version history, file audits, and folder audits.

• Navigate to the desired folder, then use the 'Upload File' menu option to upload files to that folder

• You can drag and drop files into this pop-up window or click on Browse to select files from your local directories. You can upload a single file or multiple files simultaneously
• After selecting the required files, you can verify the file names along with their sizes
• You can monitor the status of the upload in the file upload queue, and you’ll also be notified once the file is successfully uploaded. While the upload is in progress, you can continue working on other tasks simultaneously
• Verify the details and start the upload by clicking the Upload button
• To authenticate the file upload action, enable the E-signature feature in the LDAS Settings
• Authenticate the file upload action and start the upload process
• Refresh your page to view the uploaded files

Criteria to Upload File

File Upload Queue

The file upload queue lets you monitor the status and progress percentage of file uploads. You can also track manual file uploads through this queue.

From the Notifications icon, switch to the File Upload tab to check progress, remaining time, cancel uploads, or retry failed files.

Folder and file locking

Properties of a file/folder cannot be edited by multiple user accounts simultaneously. When one user is editing a folder’s properties, the folder will be locked in other users to prevent conflicting changes. The folder will remain in a locked state for others until the first user saves their changes or cancels the edit action.

You will receive a notification if you attempt to edit the properties of a file or folder that another user is currently editing. When you upload a file from a source (manual, external source, or LIID) to a designated path, the same file name cannot be uploaded from a different source, and the file version will not increment.

Upload Files

This section guides you through various methods for uploading files to Archival. Files can be uploaded to a folder in primary storage. Additionally, refer to the sections in Lab Instrument Integration on Demand (LIID) to learn how to schedule files for automatic archival.

There are three types of file uploads:

• Manual Upload
• External Source Upload
• LIID

📘

NOTE

Files cannot be uploaded to virtual folders or to folders in the secondary storage bucket.

External Source Upload

You can upload files from external source using the exposed API.

LIID

LDAS provides APIs to facilitate automatic file uploads to Archival from instruments and applications. You need to configure LIID to enable the automatic transfer of files to primary storage. With LIID, you can either upload a new file or upload a new version of an existing file.

Version Update Restriction for a File

Files in archival can be versioned by using the same file name as the existing file (file names are case-sensitive). When uploading a file to a path from any source (manual, external source, LIID), the same file name cannot be used from a different source. This constraint is in place to maintain data integrity.

Example- If a file named <File> is uploaded via LIID to the path <Path>, you cannot manually upload another file with the same name <File> to the same path <Path>.

Upload File as a New Version

To replace an existing manually uploaded file with a new version, use the same file upload procedure. When attempting to manually update the file, you will be prompted to confirm the new version update. To add a new version of an existing file, simply update the file using the same name.

Once the action is confirmed, the file will be uploaded as the latest version of the existing file. Refresh the screen to view the updated file.

Legal hold for Files/Folder

To restrict archival actions and prevent any edits, a file or folder can be placed on legal hold, which also applies to all its associated child folders and files until the legal hold is lifted. This page provides you an instruction on how to implement legal holds on files and folders.

Legal holds can only be applied to files and folders stored in primary storage. When a file or folder is placed under legal hold, all its child components (in the case of a folder) are also subjected to the hold, restricting the following operations.

• Creating subfolders
• Upload files
• Editing properties
• Archival to secondary storage

📘

NOTE

It is advisable to apply legal hold to folders that have their own archival policy, including those with inherited archival policies.

To place a file or folder under legal hold, follow these steps:

1. Utilize the 'Apply Legal Hold' menu option to place a file or folder under legal hold.

2. If E-signature is enabled in LDAS Settings, you will need to authenticate the legal hold action to complete the process.
3. A lock symbol will appear next to the file or folder, indicating that it is under legal hold.

Legal hold can be removed from a file or folder that has it applied. To do so, follow these steps:

1. Use the 'Remove Legal Hold' menu option to lift the legal hold. This option will replace the 'Apply Legal Hold' option when a file or folder is under legal hold.
2. If E-signature is enabled in LDAS Settings, you will need to authenticate the removal of the legal hold to complete the process.

After the legal hold is removed from a folder, archival actions will automatically take place if the folder has passed its archival date, during the next scheduled archival process based on the configured frequency.

When a folder is scheduled for deletion from secondary storage, it checks for all its child folders and files (if applicable) and deletes them along with the parent folder. If any child folders or files are under legal hold, the deletion process will be postponed until these items are moved to secondary storage, at which point they will be deleted along with their children.

File/Folder Activities

All actions performed within a file or folder are recorded in the File/Folder Activities. You can view these activities by navigating to the 'File/Folder Activities' page.

Troubleshooting

Mark as favourite

If you have many folders but need to access only certain ones, you can use the "Favourite" function. You can mark folders as favourites for easier access. Folders marked as favourites will display an active star next to their name. To view these folders, expand the ‘Favourites’ tab in the sidebar.

Note that only folders, not files, can be marked as favorites. You can have up to twenty folders marked as favorites at a time. If a folder is deleted after the retention period ends, it will also be removed from the favorites section.

You can remove folders from your favorites by selecting the ‘Remove from Favorites’ menu option next to the folder. Alternatively, you can use the Favorites icon in the Favorites section to remove them.

📘

NOTE

You cannot add a virtual folder to your favorites list

Follow files and folders

By following a file or folder, you can monitor it and receive alerts about any changes. This feature is available for both primary and secondary storage. You'll get in-app notifications and email alerts.

• By selecting the Follow menu option, you will receive updates through in-app notifications.

• Updates from the file or folder will appear in the ‘Notifications’ area and will be sent to you via email.

• To unfollow a file or folder, click the Unfollow menu option. This option will replace the 'Follow’ option when the file or folder is already being followed.

A followed file will trigger a notification when:

• A new version of the file is uploaded
• There are any changes in the metadata
• A legal hold is applied to or removed from the file
• The file is archived
• The file gets restored

A followed folder will trigger a notification when:

• A new sub folder is created inside that folder
• A new file is successfully uploaded in that folder
• A new version of file gets uploaded in that folder
• There are any changes in folder metadata
• Legal hold is applied or removed to/from the folder
• The folder gets archived
• The folder gets restored

You will receive a notification if any of the above actions occur in any of the subfolders or files within the followed folder.

Manage email preference

When you follow files or folders, you'll receive in-app notifications by default. If you prefer to receive email notifications also, you can turn them on by enabling the option for followed entities on the preferences page.

Search

Archival offers two types of searches. The first is a Quick search, which you can perform using the search bar. The second is the Advanced Search feature, which simplifies data retrieval by searching based on the metadata properties of files and folders.

Quick Search

This search is a free text search that allows you to locate files and folders based on various criteria. You can search for files and folders using:

1. Bucket name of the file/folder
2. Current path of the file/folder
3. Name of the file/folder
4. Type of storage in which the file/folder is stored
5. If the file/folder has any metadata associated to it, you can search using its metadata key description & list value description
6. If the file/folder has any metadata associated to it, you can search using its metadata values and list values

📘

NOTE

Currently, searching by file content is not possible, but this feature will be available in future releases.

Advanced Search

With hundreds of files being archived daily, finding specific files can be challenging for you. Metadata tags and Advanced Search can assist you in this situation. By adding context information as metadata tags to files and folders, you can enhance the search process. Advanced Search utilizes these tags to simplify data retrieval.

To maximize the effectiveness of this search, apply relevant and meaningful metadata tags to your files and folders. This approach ensures that they are easily findable by everyone, not just by project-specific personnel.

Components of Advanced Search

This section provides you an overview of the components involved in performing an Advanced Search in Archival.

Condition: Conditions form the foundation of Advanced Search. They are the search criteria that you define in the advanced search process.

A condition consists of four parts,

• The logic operators such as AND/OR for the search
• The metadata key to be searched
• The search operator to apply to the metadata key
• The value that the search operator will use

Logical Operator: Each condition is executed based on a specified logic in relation to other conditions. This logic can be either AND or OR. By default, each condition is connected using AND logic. You can select OR to change the logic as needed.

Metadata Key: The search is based on the selected metadata key, which can be of any datatype.

Search Operators: In a condition, the 'search operator' defines the criteria used for the metadata key when generating search results. The available search operators vary depending on the datatype of the selected metadata key. Based on data types, you can use search operators to refine your search results.

Query: You can combine multiple conditions into a single query, with each condition having its own AND/OR logic. A query is essentially a combination of multiple conditions. By default, AND logic is applied to a query, but you can select OR to adjust the logic as needed.

Grouping a set of conditions functions similarly to using brackets in mathematics, allowing you to nest queries within the main query.

Virtual Folder

You will see folders displayed in a virtual mode. While the original folder remains in one storage location, a virtual version will be available in another storage location to maintain the path for the child folders and files. You can easily locate this folder by clicking on the Note icon in the file explorer page.

Example

There is a folder named "Folder A," which contains a child folder called "Folder 1." After the archival period, both Folder A and Folder 1 are moved to secondary storage. If you restore only Folder 1 from secondary storage, it will be restored to primary storage, while Folder A will remain in secondary storage. When you view Folder 1, you will see Folder A listed as its parent, but you won't be able to open it.

You can also find system-defined metadata for these folders, such as the 'Linked bucket name,' in the side panel.

You can view virtual folders in the following situations:

• In the primary storage when one or more of the folder's child items are restored
• In the secondary storage when one or more of the folder's child items are archived

Troubleshooting