Function buildIndexStreaming

  • buildIndexStreaming<TypeIDType>(source, parser, pool, onRecordIndexed?, sink?): StreamingIndexResult<TypeIDType>

  • Build the entity index by streaming a ByteSource through a fixed-size moving window, instead of parsing one resident buffer. The parse loop is byte-for-byte the resident one (StepParser.parseDataBlockStreamed drives the same generator); this coordinator only owns the window: it fills it, parses the header from the first fill, then slides the window forward at top-level record boundaries as the parser advances.

    Because the parser records file-absolute addresses and its rewind stack is empty at every top-level boundary, sliding there is transparent: the unconsumed tail is copied to the front, fresh bytes are appended, and the buffer is rebased so address keeps its file-absolute value. Peak JS heap for the index build is therefore window + index columns, independent of file size.

    The window slides only once the cursor passes pool / 2, bounding the memmove frequency; a record up to pool / 2 bytes always fits after a slide. If a single record exceeds that (never on the current corpus, whose largest STEP record is ~25 KB), the whole parse restarts from the beginning with the window doubled, and repeats until every record fits — correctness over the pathological case, at the cost of a re-scan. (M1's production path will instead grow in place / restart from the last boundary; from-scratch keeps the spike simple.)

    Type Parameters

    • TypeIDType

    Parameters

    • source: ByteSource

      The byte source.

    • parser: default<TypeIDType>

      The STEP parser (typed to the schema).

    • pool: number

      Target window size in bytes.

    • OptionalonRecordIndexed: ((localID: number, expressID: number, typeID: undefined | TypeIDType) => void)

      Optional per-record event, invoked live as each top-level record is indexed with its (localID, expressID, typeID) — the seam for incremental semantic consumers (M2). localIDs are dense and assigned in parse order from 0. On the rare grow-and-restart (a single record larger than the window — never on real files, whose largest STEP record is ~25 KB), the parse re-runs from the start and records re-fire from localID 0; consumers must therefore be idempotent by localID / expressID (the standard consumers — type index keyed by localID, roots registry keyed by expressID — are). Must be synchronous and cheap; expensive work belongs on a demand queue, not the parse path.

        • (localID, expressID, typeID): void

        • Parameters

          • localID: number
          • expressID: number
          • typeID: undefined | TypeIDType

          Returns void

    • Optionalsink: StepIndexSink<TypeIDType>

    Returns StreamingIndexResult<TypeIDType>

    The index, header, result and diagnostics.

Settings

Member Visibility