Convert WordStar Documents for RAG & AI Pipelines

TL;DR

Convert WordStar documents to Markdown for retrieval and plain text for triage. Avoid PDF as an intermediate — it converts structure into layout and then makes you guess the structure back with extraction heuristics. WordStar Converter does this locally in bulk, so confidential documents never leave your network on their way into your private AI system.

The Problem: Legacy Documents Are Invisible to AI

If your organisation has been around for thirty years, a meaningful slice of its institutional memory is in formats no ingestion pipeline can read. WordStar was widely used in law firms, government departments, universities, and hospitals through the 1990s, which means founding policies, original contracts, early technical specifications, and the reasoning behind decisions still in force today may all be sitting in .ws files.

The failure mode is subtle. A retrieval system missing its historical corpus does not say "I do not have that" — it answers confidently from what it has, and the answer omits the context that explains why things are the way they are. Nobody notices, because the gap is invisible from the outside.

Step-by-Step: WordStar to Vector Database

The workflow for making a legacy .WS archive AI-ready:

Inventory Your Source Archives

Locate every .ws file, including uppercase .WS variants and versioned .wsd / .ws3–.ws8 files left behind by DOS systems. WordStar Converter scans entire folder trees recursively, so a single pass over a retired file server usually finds more than anyone expected.

Convert to Markdown (and TXT for triage)

Emit both formats in one pass. Markdown feeds the index because headings, lists, and tables survive as real structure. Plain text gives you a fast way to grep the corpus, hash for duplicates, and spot empty templates before you spend embedding budget on them.

Chunk by Heading, Not by Character Count

Fixed-size chunking cuts through the middle of arguments and separates a conclusion from its reasoning. Split on heading boundaries with a modest overlap, and carry the heading path into the chunk text so retrieval knows where each passage came from.

Attach Metadata for Filtering

Tag every chunk with source filename, document date, department, and folder path. Historical corpora make metadata filtering essential — "what did the 1996 policy say" is a filter plus a search, never a search alone.

Embed Locally

Run a local embedding model — BGE, E5, or a Sentence-Transformers variant — rather than a hosted API. If confidentiality is the reason you are building a private system, sending the corpus to a third-party embedding endpoint undoes the premise.

Load Into Your Vector Store and Query

Store vectors in pgvector, Chroma, Qdrant, or Weaviate. Instruct the model to state the date of any historical source, and inject a small glossary mapping superseded department names and product codes to current ones — recall on legacy corpora improves dramatically for almost no cost.

Format Comparison: Which Output is Best for AI?

Not all outputs are equal for LLM ingestion. Here is how the realistic targets compare:

Factor	Markdown	TXT	HTML	PDF	Raw .ws
LLM Readability	Excellent	Good	Good (tag noise)	Fair	None
Token Efficiency	High	Highest	Low (markup overhead)	Low (extraction noise)	N/A
Structure for Chunking	Real headings and tables	None	Real headings and tables	Layout-dependent	Unreadable
Tables	Pipe tables keep header/value links	Readable rows only	Real table elements	Headers detach from values	N/A
Equations	LaTeX — searchable	LaTeX or dropped	Images or MathML	Images only	N/A
Processing Complexity	Direct ingestion	Direct ingestion	HTML stripping	PDF parser / OCR	No parser exists in the stack
Best For	RAG over documents; docs-as-code	Triage, dedup, classification	Intranet publishing + crawling	Human reading, retention records	Nothing (source of truth only)

What is the best format to feed legacy documents into an LLM?

Markdown, in almost every case. It keeps the heading hierarchy your chunker needs, preserves table structure, and carries equations as LaTeX rather than as pictures. Use plain text alongside it for triage and deduplication. Avoid PDF as an intermediate — PDF extraction reintroduces running headers, hyphenation, and column-order confusion as noise the model then has to reason through.

Why Wrap Flags and Dot Commands Decide Chunk Quality

Markdown has no concept of a high-bit wrap flag — it has paragraphs and heading levels. The only way a converter can know where a sentence actually wraps, or that a line is a header from .he, is by interpreting the native WordStar stream. With that, you get real structure to chunk on. Without it, you get a flat sequence of high-bit garbage and lose the single benefit you converted to Markdown for.

So before converting an archive for a retrieval pipeline, convert a representative sample and confirm headings, page headers, and paragraph breaks survived. It takes a minute and it determines the quality of everything downstream.

Why Local Processing Matters for AI Pipelines

Most teams build private RAG systems specifically to keep regulated material — contracts, case files, personnel records, patient correspondence — off third-party infrastructure. Using a cloud-based converter to prepare those documents defeats the purpose, and it is easy to miss because conversion gets treated as plumbing rather than as processing.

WordStar Converter processes everything on your machine, so the chain of custody runs unbroken from .ws file to vector store. Pair it with a local embedding model and a self-hosted vector database and you can state honestly in a compliance review that the corpus never left the network.

How to Convert WordStar Documents for RAG Pipelines and LLM Ingestion