omnigraph/vendor/lance-table/protos/file2.proto

// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright The Lance Authors
 
syntax = "proto3";

package lance.file.v2;

import "google/protobuf/any.proto";
import "google/protobuf/empty.proto";

// # Lance v2.X File Format
//
// The Lance file format is a barebones format for serializing columnar data
// into a file.
//
// * Each Lance file contains between 0 and 4Gi columns
// * Each column contains between 0 and 4Gi pages
// * Each page contains between 0 and 2^64 items
// * Different pages within a column can have different items counts
// * Columns may have up to 2^64 items
// * Different columns within a file can have different item counts
//
// The Lance file format does not have any notion of a type system or schemas.
// From the perspective of the file format all data is arbitrary buffers of
// bytes with an extensible metadata block to describe the data.  It is up to
// the user to interpret these bytes meaningfully.
//
// Data buffers are written to the file first.  These data buffers can be
// referenced from three different places in the file:
//
// * Page encodings can reference data buffers.  This is the most common way
//   that actual data is stored.
// * Column encodings can reference data buffers.  For example, a column encoding
//   may reference data buffer(s) containing statistics or dictionaries.
// * Finally, the global buffer offset table can reference data buffers.  This
//   is useful for storing data that is shared across multiple columns.
//   This is also useful for global file metadata (e.g. a schema that describes
//   the file)
//
// ## File Layout
//
// Note: the number of buffers (BN) is independent of the number of columns (CN)
//       and pages.
//
//       Buffers often need to be aligned.  64-byte alignment is common when
//       working with SIMD operations.  4096-byte alignment is common when
//       working with direct I/O.  In order to ensure these buffers are aligned
//       writers may need to insert padding before the buffers.
//       
//       If direct I/O is required then most (but not all) fields described
//       below must be sector aligned.  We have marked these fields with an
//       asterisk for clarity.  Readers should assume there will be optional
//       padding inserted before these fields.
//
//       All footer fields are unsigned integers written with  little endian
//       byte order.
//
// ├──────────────────────────────────┤
// | Data Pages                       |
// |   Data Buffer 0*                 |
// |   ...                            |
// |   Data Buffer BN*                |
// ├──────────────────────────────────┤
// | Column Metadatas                 |
// | |A| Column 0 Metadata*           |
// |     Column 1 Metadata*           |
// |     ...                          |
// |     Column CN Metadata*          |
// ├──────────────────────────────────┤
// | Column Metadata Offset Table     |
// | |B| Column 0 Metadata Position*  |
// |     Column 0 Metadata Size       |
// |     ...                          |
// |     Column CN Metadata Position  |
// |     Column CN Metadata Size      |
// ├──────────────────────────────────┤
// | Global Buffers Offset Table      |
// | |C| Global Buffer 0 Position*    |
// |     Global Buffer 0 Size         |
// |     ...                          |
// |     Global Buffer GN Position    |
// |     Global Buffer GN Size        |
// ├──────────────────────────────────┤
// | Footer                           |
// | A u64: Offset to column meta 0   |
// | B u64: Offset to CMO table       |
// | C u64: Offset to GBO table       |
// |   u32: Number of global bufs     |
// |   u32: Number of columns         |
// |   u16: Major version             |
// |   u16: Minor version             |
// |   "LANC"                         |
// ├──────────────────────────────────┤
//
// File Layout-End
//
// ## Data Pages
//
// A lot of flexibility is provided in how data is stored.  A page's buffers do
// not strictly need to be contiguous on the disk.  However, it is recommended
// that buffers within a page be grouped together for best performance.
//
// Data pages should be large.  The only time a page should be written to disk
// is when the writer needs to flush the page to disk because it has accumulated
// too much data.  Pages are not read in sequential order and if pages are too
// small then the seek overhead (or request overhead) will be problematic.  We
// generally advise that pages be at least 8MB or larger.
//
// ## Encodings
//
// Specific encodings are not part of this minimal format.  They are provided
// by extensions. Readers and writers should be designed so that encodings can
// be easily added and removed. Ideally, they should allow for this without
// requiring recompilation through some kind of plugin system.

// The deferred encoding is used to place the encoding itself in a different
// part of the file.  This is most commonly used to allow encodings to be shared
// across different columns.  For example, when writing a file with thousands of
// columns, where many pages have the exact same encoding, it can be useful
// to cut down on the size of the metadata by using a deferred encoding.
message DeferredEncoding {
   // Location of the buffer containing the encoding.
   //
   // * If sharing encodings across columns then this will be in a global buffer
   // * If sharing encodings across pages within a column this could be in a
   //   column metadata buffer.
   // * This could also be a page buffer if the encoding is not shared, needs
   //   to be written before the file ends, and the encoding is too large to load
   //   unless we first determine the page needs to be read.  This combination
   //   seems unusual.
   uint64 buffer_location = 1;
   uint64 buffer_length = 2;
}

// The encoding is placed directly in the metadata section
message DirectEncoding {
    // The bytes that make up the encoding embedded directly in the metadata
    //
    // This is the most common approach.
    bytes encoding = 1;
}

// An encoding stores the information needed to decode a column or page
//
// For example, it could describe if the page is using bit packing, and how many bits
// there are in each individual value.
//
// At the column level it can be used to wrap columns with dictionaries or statistics.
message Encoding {
    oneof location {
        // The encoding is stored elsewhere and not part of this protobuf message
        DeferredEncoding indirect = 1;
        // The encoding is stored within this protobuf message
        DirectEncoding direct = 2;
        // There is no encoding information
        google.protobuf.Empty none = 3;
    }
}

// ## Metadata

// Each column has a metadata block that is placed at the end of the file.
// These may be read individually to allow for column projection.
message ColumnMetadata {

  // This describes a page of column data.
  message Page {
    // The file offsets for each of the page buffers
    //
    // The number of buffers is variable and depends on the encoding.  There
    // may be zero buffers (e.g. constant encoded data) in which case this
    // could be empty.
    repeated uint64 buffer_offsets = 1;
    // The size (in bytes) of each of the page buffers
    //
    // This field will have the same length as `buffer_offsets` and
    // may be empty.
    repeated uint64 buffer_sizes = 2;
    // Logical length (e.g. # rows) of the page
    uint64 length = 3;
    // The encoding used to encode the page
    Encoding encoding = 4;
    // The priority of the page
    //
    // For tabular data this will be the top-level row number of the first row
    // in the page (and top-level rows should not split across pages).
    uint64 priority = 5;
  }
  // Encoding information about the column itself.  This typically describes
  // how to interpret the column metadata buffers.  For example, it could
  // describe how statistics or dictionaries are stored in the column metadata.
  Encoding encoding = 1;
  // The pages in the column
  repeated Page pages = 2;   
  // The file offsets of each of the column metadata buffers
  //
  // There may be zero buffers.
  repeated uint64 buffer_offsets = 3;
  // The size (in bytes) of each of the column metadata buffers
  //
  // This field will have the same length as `buffer_offsets` and
  // may be empty.
  repeated uint64 buffer_sizes = 4;
} // Metadata-End

// ## Where is the rest?
//
// This file format is extremely minimal.  It is a building block for
// creating more useful readers and writers and not terribly useful by itself.
// Other protobuf files will describe how this can be extended.