Archival

The LDAS Archival module provides an all-in-one solution for managing scientific data in the AWS S3 cloud. Specifically designed for the scientific community, it meets regulatory standards and is GxP compliant, ensuring your data management is secure and compliant. This module allows you to store and manage data from instruments and applications along with metadata over the long term.

Archival offers both high-performance and low-performance storage options, allowing you to organize data based on how often it’s used. By keeping frequently accessed files separate from those that are rarely used, you can reduce storage costs over time. Additionally, metadata tagging, and advanced search features simplify locating and managing your data efficiently.

Key Terms

This section provides an overview of the terminologies used in Archival.

Storage containers: These containers hold archival files and folders, organizing the retention and movement of archival data. There are two types of storage containers namely primary storage and secondary storage.

1. Primary storage

Primary storage includes high-performance, high-cost containers where you can create folders and upload files. It also stores files from Lab Instrument Integration on Demand (LIID) that are scheduled for automatic archival. You can add more details to these files using metadata tagging.

2. Secondary storage

Secondary storage is a low-cost backup container meant for projects that aren’t accessed frequently. This lower-performance option helps reduce storage costs by letting you schedule and move rarely accessed files here.

📘

NOTE

Files cannot be uploaded directly to secondary storage, as it operates in read-only mode. Files and folders are automatically moved from primary storage to secondary storage once their archival period ends.

Archival Policy

The Archival Policy section contains metadata related to archival conditions, outlining the phases of a folder's lifecycle in Archival. Archival policies cannot be applied to files. The following are the key metadata elements within the archival policy:

• End Date
• Archival Period
• Archive Date
• Retention Period
• Deletion Date

In LDAS, you can set values for metadata like End Date, Archival Period, Retention Period, and Restore Period to define how long folders are held in Archival. Other metadata values, such as Archive Date, Deletion Date, and Restoration End Date, are automatically calculated based on these settings.

Archival policy can only be set at the folder level. You can define these values when creating a folder or update them later. While the folder is in primary storage, you can adjust its archival policy as needed. The policy values set for a folder will be inherited by all the folders and files inside it.

The diagram below will help you understand the archival policy,

📘

NOTE

Child folders can have a different archival policy than their parent folder. If a child folder does not have explicitly defined archival policy values, it will inherit the archival policy values from its parent folder.

End Date

The End Date signifies the completion date of the project associated with the folder. Both the archival and retention periods are calculated from this date.

You can set the End Date to the past, present, or future at the parent folder and any child folders. However, the End Date of a child folder cannot be set to a date later than that of its parent folder.

Example :

In Program 1, you may have two projects, such as Project A and Project B. The end date can be set at the Program 1 level, assigning the same end date to both projects. Alternatively, you can set an earlier end date specifically for Project B if needed.

Archival Period

The Archival Period specifies the duration for which the folder should be stored in primary storage. This period can be defined in terms of days, weeks, months, or years.

In Archival, the calculation of weeks, months, and years in terms of days is as follows:

• 1 week = 7 days
• 1 month = 30 days
• 1 year = 365 days

📘

NOTE

It is recommended to assign longer archival periods to frequently accessed and referenced folders, ensuring they remain in higher performance primary storage for extended durations.

Retention Period

The Retention Period defines how long an archived folder should remain in secondary storage before permanent deletion. You can set this period in terms of months or years.

Archival Date

Based on the provided End Date and Archival Period, LDAS calculates the Archival Date of a folder as 'End Date + Archival Period.' Once the Archival Date is reached, the folder and all its child folders will be moved from primary storage to secondary storage in the next archival schedular run after the archival period.

For each instance of LDAS, there is a fixed frequency at which folders and their child folders are moved to secondary storage if their Archival Date has passed before this frequency.

To restrict archival actions and prevent any edits, a file or folder can be placed on legal hold, which also applies to all its associated child folders and files until the legal hold is lifted.

📘

NOTE

Typically, the archival scheduler runs on a weekly or monthly basis, depending on your chosen settings.

Example

If a folder has an 'End Date' of June 23, 2023, an 'Archival Period' of 1 month, and a 'Retention Period' of 2 months, it will move from primary to secondary storage during the fixed frequency run that occurs after the completion of the Archival Date, which is July 23, 2023.

After archival, a system-defined metadata key 'Previous Bucket Name' will appear in the side panel. This key contains information about the primary container to which the folder belonged before it was archived.

This metadata key will be available for files and folders that have been moved or archived to secondary storage . You can also restore files if you need to move them to high-performance primary storage.

Deletion Date

Using the provided End Date and Retention Period, LDAS calculates the Deletion Date of a folder as 'End Date + Retention Period.' After the Deletion Date, the folder and all its child folders will be permanently deleted from secondary storage and thus removed from LDAS in the next retention scheduler run after the deletion date.

For each instance of LDAS, there is a fixed frequency at which folders and their child folders are deleted from secondary storage if their Deletion Date has passed before this frequency.

To restrict archival actions and prevent any edits, a file or folder can be placed on legal hold, which also applies to all its associated child folders and files until the legal hold is lifted.

📘

NOTE

Manual deletion of files and folders in Archival is not supported. To expedite the removal of a folder from storage, reduce its Archival and Retention Periods in the archival policy.

Example

If a folder has an 'End Date' of June 23, 2023, an 'Archival Period' of 1 month, and a 'Retention Period' of 2 months, it will be deleted from secondary storage during the fixed frequency run that occurs after the completion of the Deletion Date, which is August 22, 2023.

Criteria for Valid Archival Policy

Restore Period

Once a folder moves to secondary storage, it becomes non-editable. However, if you need to edit values, upload files to an archived folder or if you need to frequently access the folders or files, you can restore it to primary storage using the restore option.

When you restore a folder, all its child folders will move back to primary storage, making them editable again. However, the existing End Date and Archival Period cannot be modified for the restored folder or files.

For restored files, the Retention period and Restoration period cannot be edited on the properties page.

Restoration End Date

Using the provided Restore Period, LDAS calculates the Restoration End Date as 'Date of Restoration + Restore Period.' The Date of Restoration is the date when the folder is restored. Once the Restoration End Date is reached, the folder will be moved back to secondary storage from primary storage. However, if the folder has already reached its Deletion Date, it will be permanently deleted from the system instead.

Criteria for Valid Restore Period

A Restore Period is considered valid only if it satisfies the following conditions:

• It cannot exceed the Deletion Date of the folder
• It cannot exceed the Restoration End Date of the parent folder in primary storage (applicable only if the parent folder is restored before the child folder)
• It should be greater than or equal to the Restoration End Date of any child folder that was restored before the folder in question

Example

Consider the following folder path: Folder A (Parent) -> Folder B (A’s Child) -> Folder C (B’s Child)

Let the archival policy of the folders be as follows:

Consider if the fixed frequency for archival occurs daily at 12 AM in UTC time zone and the fixed frequency for retention occurs daily at 12 PM in UTC time zone.

For the archival policies mentioned, by 31-Oct-2023, all folders will be in secondary storage. Then,

Therefore, when restoring a folder, the restore period will be limited by the archival policies of both the parent and child folders.

Restrictions for Restoration

When you are scheduling a folder for restoration, other person cannot simultaneously schedule restoration for that folder or any folder within its hierarchy, whether above or below it.

In these cases, the restore action for the folder will be locked for other person until the folder either moves back to primary storage or the action is cancelled.

To restrict archival actions and prevent any edits, a file or folder can be placed on legal hold, which also applies to all its associated child folders and files until the legal hold is lifted.