Files-11

The native OpenVMS file system comes from older DEC operating systems and is very similar to them in many ways. The main difference is the location of the directories. These file systems provide some form of rudimentary non-hierarchical directory structure, usually based on allocating one directory to one user account. In RSTS / E, each user account was represented by two numbers, a pair [project.programmer], and had one associated directory. Special system files, such as program executables and the OS itself, were stored in a directory reserved for the system account.

Although this was applicable on PDP-11 systems with limited storage capacity, VAX systems with much larger hard drives required a more flexible file storage method: in particular, hierarchical directory layout, the most notable improvement in ODS-2.

Files-11 is the common name for five separate file systems, known as levels 1 through 5 of the on-disk structure (ODS).

ODS-1 - a flat file system used in RSX-11; It is supported by older versions of VMS for compatibility with RSX-11, but has never been used by VMS itself; was almost universally replaced by ODS-2 and ODS-5.

ODS-2 - the standard file system for VMS, still remains the most frequently used file system for the system disk (the disk on which the operating system is installed).

Although they are rarely mentioned by their ODS level designation, ODS-3 and ODS-4 are Files-11 support for the ISO 9660 and High Sierra CD file systems, respectively.

ODS-5 is an extended version of ODS-2 available for Alpha or Itanium platforms , which adds support for persistent register of file names with non- ASCII characters and improved support for hierarchical directories. Initially, it was intended to serve files in Microsoft Windows or other systems other than VMS, as part of the NT affinity project, but it is also used on user disks and Internet servers .

All files and directories in the Files-11 file system are located in one or more parent directories, and finally in the root directory, the master file directory (MFD). Consequently, the file system is organized into a tree structure.

In the example on the right, File 2 has an entry in the Dir 2 and Dir 3 directories; It is in both directories at the same time. Even if it is removed from one directory, it will still exist in another until it is removed from there. This is very similar to the concept of hard links on UNIX , although care must be taken not to delete the file on disks that do not support hard links. Hard links are only available on ODS-5 drives, and only if they are allowed on those drives.

A functioning VMS system has access to one or more permanently connected drives, each of which contains a fully-fledged independent file system. They are either local storage, or, in the case of a cluster , storage shared with remote systems.

In the OpenVMS cluster configuration, non-private disks are shared by all cluster nodes (see Figure 1). In this configuration, two system disks are available for both cluster nodes, but the private disk is not shared: it is connected only for use by an individual user or process on the first machine. Access to files in the cluster is controlled by OpenVMS Distributed Lock Manager, an integral part of the file system.

Multiple drives can be combined into one larger logical drive or a combined volume (volume set). Disks can also be automatically copied to shadow sets to protect data or better read performance.

A disk is designated either by its physical name or (more often) by its logical name as defined by the user. For example, a boot device (system disk) may have a physical name of $ 3 $ DKA100, but it will usually be referenced by the logical name SYS $ SYSDEVICE.

The file system on each disk (except for ODS-1) is hierarchical. The full file specification consists of the host name, user account name and password, device name, directory, file name, file type and file version and has the following format:

  NODE "accountname password" :: device: [directory.subdirectory] filename.type; version

  user password node :: device: [directory] file_name. file type; version

For example, [DIR1.DIR2.DIR3] FILE.EXT will refer to the latest version of the FILE.EXT file on the default drive in the [DIR1.DIR2.DIR3] directory.

DIR1 is a subdirectory of the main file directory (MFD), or root directory, and DIR2 is a subdirectory of DIR1. The main file directory is defined as [000000].

Many parts of the file specification can be omitted, then they are taken from the current file specification by default. The default file specification replaces the concept of the “current directory” on other operating systems by providing a set of default settings for the node, device name, and directory. All processes have a default file specification, which contains the drive name and directory, and most VMS file system routines accept a default file specification, which can also contain a file type; the TYPE command, for example, expects “.LIS” as the file type by default, therefore, if the TYPE command specifies the file name F without the extension, it will try to open the F.LIS file.

Each file has a version number, which by default is 1 if there are no other versions of this file (otherwise, it is one more than the highest version). Each time you save or overwrite an existing version, a new file will be created with the same name but increased by 1 version number. Older versions can be deleted explicitly with DELETE or PURGE, or optionally, older versions of a file can be deleted automatically when the file version limit is reached (set by the SET FILE / VERSION_LIMIT command). In this case, older versions are not overwritten, but stored on disk and can be restored at any time. The laid down limit of the version number is 32767. The operating mode of the versioning mechanism can be easily disabled if it is not necessary. In particular, files that are updated directly, such as databases, do not create new versions, unless specifically programmed.

ODS-2 is limited to 8 levels of directory nesting, and file names are only in upper case, alphanumeric names (plus an underscore, dash, and dollar sign) up to 39.39 characters (39 for the file name and 39 for the extension). In ODS-5, the character set is expanded to lowercase and most other ASCII characters, as well as ISO Latin-1 and Unicode characters , the maximum file name length is increased, and the number of directory nesting levels is unlimited. When constructing the full file name for the ODS-5 file, which uses unresolved characters in ODS-2, the special syntax “^” is used to maintain backward compatibility; the file “file.tar.gz; 1” on the ODS-5 disk, for example, will be called “file ^ .tar.gz” - the file name will be “file.tar” and the extension “.gz”.

File protection in VMS is determined by two mechanisms: access control based on user identification code (UIC) and access control based on access control lists. Access control based on the user identification code is based on the owner of the file and its identification code, or the identification code of the user who wants to access the file. Access is defined by four permission groups:

• System Permissions
• Owner Permissions
• Group permissions (Group)
• World Permissions

And with four permission bits:

• Read
• Write
• Execute
• Delete

System access is applied to any user whose identification code is less than or equal to the SYSGEN MAXSYSGROUP parameter (usually 8 or 10 octal) (for example, SYSTEM user); owner and group access applies to the owner of the file and its group to which it belongs, and world access applies to any other user. There is also a fifth permission bit - Control, which is used to determine access to change file metadata such as protection. This group cannot be specified explicitly; it is always set for the System and the Owner, and never set for the Group or the World.

Access control based on the identification code is also influenced by four system privileges that allow them to be held to prevent access control interception:

• BYPASS: the user implicitly has RWED access to all files, regardless of file protection;
• READALL: the user implicitly has access on R to all files;
• SYSPRV: the user can access files based on System protection;
• GRPPRV: a user can access files based on System protection if their UIC group reaches the file group.

ACL allows you to assign additional privileges to a user or group; for example, a web server user identification code may be given the right to read all files in a separate directory. ACLs can be marked as inherited, where the file ACLs for the directory are applied to all files in it. Changes to access control lists are made by the EDIT / ACL command and take the form of an identifier / access_pair pair. For example, writing to an ACL command

  (IDENTIFIER = HTTP $ SERVER, ACCESS = READ + EXECUTE)

allows the HTTP user $ SERVER to read and execute files.

A logical name is a system variable that can refer to a disk, directory, or file, or contain other program information. For example, the logical name SYS $ SYSDEVICE contains the system boot device. Logical names usually refer to a single directory or drive, for example, SYS $ LOGIN, which is the home directory (s) of a user account; these logical names cannot be used as real disk names - SYS $ LOGIN: [DIR] FILE is not a valid file specification. However, hidden logical names defined by the DEFINE / TRANSLATION = CONCEALED command can be used for this; these root directories end with a period character (".") in the directory specification, so the command

  $ DEFINE / TRANS = CONCEAL HOME DISK $ USERS: [username.]

will allow to use HOME: [DIR] FILE. More common are simple logical names that point to specific directories associated with any application software that can be located on any disk or in any directory. Therefore, the logical name ABC_EXE can point to the ABC application executable directory, and ABC_TEMP can point to the temporary files directory for the same application and this directory can be on the same drive and in the same directory as ABC_EXE, or it can be anywhere on another drive (and in a different directory tree).

Logical names do not have close equivalents on POSIX- compliant operating systems. They are similar to UNIX environment variables , except that they are extended by the file system, not by the shell or application. They must be defined before use, so they are common to many logical names defined in the system batch file of automatic startup, as well as in batch files of user accounts.

The closest non-VMS operating system that supports the concept of logical names is AmigaOS , through the ASSIGN command. The AmigaDOS disk operating system included with AmigaOS seems to have taken a lot from VMS, given that TRIPOS (whose port is AmigaDOS) was created under the strong influence of VMS. For example, the names of physical devices follow a pattern like DF0: for the first floppy drive, CDROM2: for the third CD-ROM drive, etc. However, since the system can boot from any connected drive, the operating system creates a logical SYS name: assigned automatically referring to the boot device in use. Other assignments, LIBS :, PREFS :, C :, S: and others are also created that do not refer to SYS :. Users, of course, are also allowed to create and delete their own appointments.

Logical names can refer to other logical names (up to a predefined nesting limit of 10) and can contain lists of names to search for existing file names. Some frequently referencing logical names are:

Record Management Services (abbreviated as RMS, rus. Record Management Services) is the structural level of input-output of the VMS operating system. RMS provides comprehensive software support for managing structured files, such as record files and indexed database files. A VMS file system can be represented as a database containing a series of records, each of which has one of many individual fields. A text file, for example, is a list of records (lines) separated by a newline. RMS is an example implementation of a record-oriented file system.

There are 4 record formats defined by RMS:

• Fixed length - all entries in the file are the same length.
• Variable length - records vary in length, and each record is preceded by a byte counter giving its length.
• Variable length record with a fixed control unit - records vary in length, but are preceded by a fixed length control unit.
• Stream - records vary in length, and each record is separated from the next by a trailing character. A text file is an example of a stream format file that uses end-of-line or carriage return characters to separate records.

There are 4 methods for accessing records, or a method for finding existing records from files:

• Sequential access - initially with separate records, subsequent records are in order to the end of the file.
• Relative access by record number - record is located by its number relative to the beginning of the file.
• Access to the file record address — records are located directly by their location in the file (RFA - Record File Address, Russian file record address).
• Index access - records are found by key, in the form of a key-value correspondence (associative array).

At the disk level, ODS represents the file system as an array of blocks; a block consists of 512 bytes located on a single physical disk (volume). Disk blocks are assigned to clusters (initially 3 continuously located blocks, but later increased for larger disk sizes). Ideally, the file on the disk should be completely continuous, that is, the blocks containing the file should be arranged sequentially, but fragmentation of the disk will sometimes require placing the file in inconsistent clusters, in which case fragments will be called “extents”. Disks can be combined with other disks to form a unified volume, and files are stored anywhere on the entire set of disks, but the large size of the disks reduces the use of unified volumes, because managing single disks is easier.

Each file on a disk (or unified volume) with the Files-11 file system has a unique file identifier (FID) consisting of 3 numbers: file number (NUM), file sequence number (SEQ) and relative volume number (RVN). NUM shows where the INDEXF.SYS file containing the file metadata is located; SEQ - the generation number that increases when the file is deleted and another file is created repeatedly using the same entry in INDEXF.SYS (therefore, any dangling links to the old file will not accidentally point to a new one); RVN shows the number of the volume on which the file is stored when the merged volume is used.

Structural support for the ODS volume is provided through a catalog file — a special file containing a list of file names, file versions, and associated file identifiers (FIDs). The root of the directory structure is the main file directory (MFD), the root directory that contains (directly or indirectly) each volume file.

(picture)

This diagram shows an example of a directory containing 3 files and how each file name is displayed in an entry in INDEXF.SYS (each INDEXF entry contains more information, only the first few items are shown here).

At the top level of the ODS file system is the main file directory (MFD), which contains all the directory files of the highest level (including itself) and several system files used to store file system information. On volumes with ODS-1, a two-level directory structure is used: each user identification code (UIC) is associated with a user file directory (UFD) in the form [GROUP.USER] ([GROUP USER]). On ODS-2 and later volumes, the location of directories in the main file directory is arbitrary, being limited by the nesting limit of directories (8 levels in ODS-2 and unlimited in ODS-5). On merged volumes, the main file directory is usually stored on the first volume and contains subdirectories of all volumes.

The following system files are presented in the main ODS file directory:

• INDEXF.SYS; 1 - index file
• BITMAP.SYS; 1 - data warehouse bitmap file
• BADBLK.SYS; 1 - bad blocks file
• 000000.DIR; 1 - directory file of the most basic file directory
• CORIMG.SYS; 1 - kernel image file
• VOLSET.SYS; 1 - volume list file of the merged volume (only in ODS-2/5)
• CONTIN.SYS; 1 - continuation file (ODS-2/5 only)
• BACKUP.SYS; 1 - backup log file (only in ODS-2/5)
• BADLOG.SYS; 1 - pending bad blocks (only in ODS-2/5)
• SECURITY.SYS; 1 - volume security profile (only in ODS-2/5)
• QUOTA.SYS; 1 - quota file (optional and available only in ODS-2/5)
• GPT.SYS; 1 - GUID Partition Table (GPT) (EFI boot structures for OpenVMS I64, optional for OpenVMS Alpha)

Remember that the file system implementation itself does not refer to these files by name, but by their file identifiers, which always have the same value. So, for example, INDEXF.SYS is always a file with NUM = 1 and SEQ = 1.

The index file contains the most basic information about the unified volume Files-11.

There are two ways to organize INDEXF.SYS, the traditional organization and the organization used on GPT.SYS disks with GUID Partition Table Structures (GPT).

In a traditional organization, block 1 is a boot block that contains the location of the primary boot image used to boot the VMS operating system. It is always located on disk in logical block 0, so the hardware firmware can read it. This block is always there, even on non-system (non-boot) volumes.

After the boot block, the primary home block is located. It contains the name of the volume, the location of the extents, including the rest of the index file, the user identification code (UIC) of the owner of the volume, and information about the protection of the volume. Usually there are several additional copies of the home block, known as secondary home blocks, which allow you to restore if it is destroyed or damaged.

Disks containing GPT.SYS contain the equivalent of a boot block (known as the Master Boot Record (MBR)) and they do not have a primary home block. All home blocks presented on the GPT disk are alternative home blocks. These structures are not included in INDEXF.SYS and blocks from the INDEXF.SYS file are not used.

The remainder of the index file consists of the file headers that describe the extents where the file is located in the volume, and the file metadata, such as the user identification code (UIC) of the owner, access control lists, and security information. Each file is described by one or more file headers — more than one may be required when the file has a large number of extents. The file header is a fixed-length block, but contains sections of both fixed and variable lengths:

• The header section contains NUM and SEQ, protection (security) information and the location of the remainder of the file header.
• The ident section contains account metadata: file name, creation and modification times, and last backup time.
• The map section describes which blocks (extent) of the physical disk are mapped to each virtual block of the file.
• The access control list section contains access control list information about the file.
• The reserved area section is the place at the end of the file header that is not used by the operating system. It can be used for supplier or consumer information.
• The last 2 bytes of the file header are the checksum of the previous 255 words to verify the header is correct.

If possible, the map and ACL sections of the header file are contained entirely in the primary header (if there are several). However, if the ACL information is too long or the file contains too many extents, there will not be enough space in the primary header to store it. In this case, an extended header is allocated to store the remaining information.

(picture) INDEXF.SYS file header structure

The file header begins with 4 offsets (IDOFFSET, MPOFFSET, ACOFFSET and ROFFSET). Since the size of the areas after the fixed-length headers can vary (areas such as map and ACL), offsets are required to locate these additional areas. Each offset is equal to the number of 16-bit words from the beginning of the file header to the beginning of the area for which the offset is calculated.

If the file requires multiple headers, then the segment extension (SEGNUM) contains the sequence number of this header, starting with 0 in the first entry in INDEXF.SYS.

STRUCLEV contains the current structure level (in high byte) and the version (in low byte) of the file system; ODS-2 has a level of structure 2. An increase in the version number indicates a backward compatible change, which may be ignored by older software; changes in the structure level itself are incompatible.

W_FID (contains 3 values: FID_NUM, FID_SEQ and FID_RVN, respectively, file, sequence and relative volume number) contains the identifier of this file; EXT_FID (also consisting of 3 values) holds the location of the next extended header, if any. In both of these values, RVN is defined as 0 to represent the “current” volume (0 is normally not a valid RVN).

FILECHAR contains several flags that specify how the file is processed or organized:

• NOBACKUP sets that the file will be ignored when performing the backup.
• WRITEBACK allows cached (deferred) write to a file.
• READCHECK establishes that all file reads are done twice and compared to ensure data integrity.
• WRITCHECK - all records will be checked by subsequent reading and comparison.
• CONTIGB - the operating system will try to place the file in the storage as consistently as possible.
• LOCKED - set if the file is locked at the end of access. If installed, this indicates that the file has not been properly closed since its last use, and the contents may be inconsistent.
• CONTIG indicates that the file is stored on disk in sequence; that is, each virtual block i is mapped to a logical (physical) block i + k, where k is a constant
• BADACL - set if the file has the wrong access control list.
• SPOOL - installed if the file is spooled, as an intermediate file used in the printing process.
• DIRECTORY - installed if the file is a directory.
• BADBLOCK - installed if the file contains bad blocks.
• MARKDEL - installed if the file was marked for deletion but is still in use; it will be deleted immediately after closing by the last user.
• NOCHARGE, if set, indicates that the place used by the file is not taken from the storage owner quota.
• ERASE - indicates that the contents of the file are overwritten when it is deleted.

The ACCMODE flag describes the privilege level with which the process can be executed in order of access to the file. VMS defines 4 privilege levels: user, supervisor, exec, and kernel. Each type of access — read, write, execute, and delete — is encoded with a 2-bit integer value.

The FILEPROT flag contains discretionary access control information applicable to the file. It is divided into 4 groups of 4 bits each: system, owner, group and world. Bit 0 corresponds to read access, bit 1 to write, bit 2 to execute, and bit 3 to delete. Setting a bit prohibits partial access to the group; removal of a bit - allows.

If the file header is an extended header, then BACKLINK contains the file identifier from the main header; otherwise, it contains the identifier of the directory file containing the master record of the file.

• Storage Bitmap File: BITMAP.SYS

The bitmap file is responsible for storing information about the used and available place in the volume. It contains a storage control block (SCB), which includes summary information and a bitmap — an array of bits indicating whether a cluster of blocks on a disk is free or distributed. In early versions of VMS, the cluster consisted of 3 blocks, but as disk sizes increased, so did the cluster size.

• Bad Blocks File: BADBLK.SYS

The bad blocks file contains a list of bad blocks on the physical volume, so the system can avoid distributing them to files. This file was used more in the early days when disks were usually produced with a lot of bad patches on the surface.

• Merged Volume Volume List File: VOLSET.SYS

The volume list of the federated volume is located on the first volume of the federated volume and contains the label list of all volumes in the set and the name of the volume set.

• Continuation File: CONTIN.SYS

When a file is located on a multi-volume set, crossing the boundaries of two separate volumes, the continuation file is used as its extended header and tells where to find the rest of the file.

• Quota file: QUOTA.SYS

The quota file contains information about disk quota usage by each UIC. Contains an entry for each user identification code with a place allocated to it in the volume, in accordance with the information how much space can be used by this UIC. Note: the disk quota feature is optional and the file will exist only if this feature was enabled.

• Volume Security Profile: SECURITY.SYS

The volume security profile contains the owner’s UIC of the volume, a facemask, and volume access control list.

• GUID Partition Table: GPT.SYS

This file overlaps and protects the MBR (Master Boot Record) and GPT (GUID Partitioning Table) disk structures used for EFI-compatible firmware. This file is created by default during disk initialization in OpenVMS I64, and is created optionally (with the INITIALIZE / GPT command) in OpenVMS Alpha.

• File system list
• File System Comparison

1. Andrew C. Goldstein, VAX / VMS Software Development (1985-01-11). Files-11 On-Disk Structure Specification.
2. Hewlett-Packard Development Company, LP (September 2003). "Appendix A: Files-11 Disk Structure." OpenVMS System Manager's Manual, Volume 2: Tuning, Monitoring, and Complex Systems.
3. Kirby McCoy (1990). VMS File System Internals. Digital Press. ISBN 1-55558-056-4 .

• OpenVMS Documentation: OpenVMS File Guide
• http://www.vms2linux.de