Development Documentation

Architecture Overview

Key Concepts

Data

Objects

Relationships

Attributes

Folders

Keys

Model

Objects Types

Relationship Types

Attribute Types

Attribute Definitions

Packages

Administration

Users and Teams

ACLs

Profiles

Repository

Files and Vaults

Business Objects

BO Implementation

BO Inheritance

Transactions

Queries

Query Architecture

Query Specifications

Find Queries

Expand Queries

Join Queries

Services

Service Configuration

Service Builder

Logging

Unit Testing

XML

ice.NET Studio

ice.NET SDK

ice.NET Web SDK

Tutorials

Reference Documentation

Knowledge Base

 

ice.NET > Developers > Development Documentation > Key Concepts > Files and Vaults

ice.NET Key Concepts: Files and Vaults

Files and File Content

The built-in object type Core.File (display name: File) provides the functionality of storing unstructured content. All instances of Core.File or of one of its subtypes manage a ordered sequence of file versions. Each Version points to exactly one FileContent. The FileContent record in the database represents a chunk of binary data that is identified by a UUID and that is stored in a place determined by the type and configuration of its containing vault.

The structure and cardinalities are as follows:

The diagram indicates that multiple versions of the same or different instances of Core.File can share the same FileContent. This is especially useful for copying an object instance without increasing the vault space required to store the binary content.

Vaults

Vaults are places where unstructured file content is stored. Each vault has a unique name in the scope of an ice.NET system and each file content stored in the vault has a universally unique identification (UUID). Vaults can be created (and modified and deleted) using the standard administration tools (ice.NET Studio SmartClient, Web UI) and the Vault API represented by the CLR interface PDTec.IceNet.Core.Database.IVault.

There are several types of vaults, mainly separated in two categories:

• Database vaults
• File vaults

Database Vaults

Database vaults store their file contents – as the name suggests – in the same database that also contains the structured data (objects, relationships and attributes).

Advantages:

• No configuration necessary (and possible).
• Vault access takes place within database transaction boundaries, always 100% consistent.
• Backup of the vault content is automatically included in every database backup.

Disadvantages:

• Vault content increases database size.
• Requests to vault content always have to go through the application server, vault has no separate interface.

File Vaults

The category of file vaults represents all other types, i.e. all types of non-database vaults. The Version and FileContent records are still stored in the database for consistency reasons but the actual chunk of binary data is stored elsewhere.

Advantages:

• The vault content does not add to the database size.
• The storage and access methods can be separated from the application server and delegated to a specific vault service in order to reduce the network load on the application server due to file transfer.

Disadvantages:

• Filesystem vaults require some configuration.
• Vault access is separated from database transactions. This can lead to inconsistencies between vault content and database content due to access failures and transactions that were rolled back. The ice.NET implementation ensures that in case of failure no checked-in file is lost but it cannot technically avoid that unreferenced binary content (“dead data”) remains stored under certain circumstances. A vault admin command (“tidy”) helps to detect and remove dead data from the vault periodically.
• Database backup is not sufficient to save the state of an ice.NET system. The vault directories must be backed-up as well.

The term “filesystem” vault is for historical reasons (and the fact that ultimately the binary data is store somewhere on a file system). However, technically “filesystem” means that the binary data is stored outside the database and accessed through a vault protocol. The methods of the ice.NET vault protocol are defined in the CLR interface PDTec.IceNet.Core.Database.IVaultProtocol. Any valid implementation of this interface can serve as a vault and can be plugged into an ice.NET system by configuration.

Example 1: Working with Files

This example explains how to use the PDTec.IceNet.Core.Model.File.IFile interface to read and write file content.

The following code shows how to create a file object and to store binary content into the built-in vault System.Database:

IObjType pFileObjType = m_pRepository.GetObjTypeByName("Core.File");
byte[] aContent = Encoding.ASCII.GetBytes("FILECONTENT");

Repository.ExecuteTransaction(delegate()
{
    // Create a file object...
    IFile pFile = pFileObjType.Create<IFile>(m_pRepository.RootFolder, "F1", "");
    
    // Store content (implicitly creates a first content version)......
    pFile.WriteContent(true, "", "System.Database", "f1.txt", aContent);
});

The following code shows how read binary content from a file:

IFile pFile = Repository.Get<IFile>(objectId);
byte[] aContent;

// Reads file content from the current (most recent) file version into buffer...
pFile.ReadContent(out aContent);

// Gets the file size of the current version...
long size = pFile.GetContentSize();

// Reads file content with offset and partial size parameters...
pFile.ReadContentPart(0, (int)size, out aContent);

The following code shows how to use the AppendContent function to subsequently add content to a file:

IObjType pFileObjType = m_pRepository.GetObjTypeByName("Core.File");

byte[] aContent0 = Encoding.ASCII.GetBytes("F");
byte[] aContent1 = Encoding.ASCII.GetBytes("1");

Repository.ExecuteTransaction(delegate()
{
    // Create a file object...
    IFile pFile = pFileObjType.Create<IFile>(m_pRepository.RootFolder, "F1", "");
    
    // Store initial content (implicitly creates a first content version)...
    pFile.WriteContent(true, "description", "System.Database", "f1.txt", aContent0);
                        
    // Append content to the current version... 
    pFile.AppendContent(aContent1);
});

Example 2: Working with Vaults and FileContent

This example explains how to work with vaults and file content directly. This can be useful to implement a transactional safe Web uploading process that enables to upload large files in parts, where the whole uploading process cannot be encapsulated in a single database transaction for HTTP request memory size and timeout reasons:

• Create a FileContent record in the database and retrieve its UUID.
• Upload the file in several steps by appending to the FileContent.
• Create the File object and connect the FileContent.

This procedure ensures that if the upload process fails at one point, the file object does not appear in the database at all. This avoids broken file objects. Unconnected FileContents can be identified and removed easily by a vault utility.

The following code shows how to create a file in several steps and how the individual transactions can be separated from each other. If the vault is not a database vault, the transaction scopes around the IVaultProtocol calls are not necessary.

byte[] aContent = Encoding.ASCII.GetBytes("1234567890");

IVault pVault = Repository.GetVaultByName("System.Database");

string contentUuid = null;

//
// First step: create a FileContent record in the database...
//

string hash = FileUtils.CalculateHash(aContent);

Repository.ExecuteTransaction(delegate
{
    contentUuid = pVault.CreateFileContent(aContent.Length, hash).Uuid;
});

//
// Second step: Use the IVaultProtocol of the vault to upload the content...
//

IVaultProtocol pProtocol = pVault.GetProtocol();

Repository.ExecuteTransaction(delegate()
{
    pProtocol.CreateFileContent(contentUuid, aContent.Take(5).ToArray());
});

Repository.ExecuteTransaction(delegate()
{
    pProtocol.AppendFileContent(contentUuid, aContent.Skip(5).ToArray());
});

//
// Final step: Create the File object and connect the FileContent as a new content version...
//

IFile pFile = null;

Repository.ExecuteTransaction(delegate
{
    pFile = Repository.GetObjTypeByName("Core.File").Create<IFile>(Repository.RootFolder, "F1", "");

    pFile.ConnectContent(true, "", "test.txt", contentUuid);
});

---

Using Files in the application model

The Core.File object type provides the basic content management functionality. It is recommended to define application-specific document object types to reflect the actual usage of this content management functionality. The following image suggests two ways to implement domain-specific document types:

In the MyApp model the Core.File type is used as a supertype of a domain-specific type MyApp.Attachment. The MyApp.Attachment inherits the content management functionality and can additionaly define attributes and BO methods to extend the Core.File functionality in a domain-specific way.

In the YourApp model a more complex document type is created that is not a subtype of Core.File but instead owns 1:n files. A reason for this model could be that one logical document contains several binary representations in different formats. The business logic of YourApp.Document is now responsible for managing the collection of files attached to the document object.