Whatever message this page gives is out now! Go check it out!
embedding-model-configuration.md and your Milvus, Pinecone, Qdrant, or Chroma vector store docs).Field | Description |
|---|---|
Vector store | Required. The target vector store profile where ingested chunks (and embeddings) are stored. Choose a store that matches your RAG query path and embedding dimension. If the dropdown is empty, create and save a vector store configuration first. |
Field | Description |
|---|---|
Source type | Single file — ingest one document by path. Directory — ingest all supported files under a folder (respecting your product’s recursion and filter rules). URL — fetch content from a web address when your product supports it. |
File path (or path / URL field) | Path or address for the selected source type. For single file or directory, use an absolute path the server can read. Use Browse server when available to reduce path typos. For URL, enter a full URL per your integration’s requirements. |
Supported formats | Typical support includes PDF, Word, Excel, PowerPoint, HTML, CSV, JSON, XML, plain text, Markdown, and related formats—exact list depends on your release. Unsupported files may be skipped or fail per Continue on error. |
Field | Description |
|---|---|
Parser type | Format-specific parser (for example PDF, HTML, plain text). Choose the parser that matches the dominant file type in this run, or the type your product uses when a single parser is selected for a batch. Some products auto-detect per file; confirm behavior in your docs. |
Character encoding | Text encoding for parsers that read byte streams (for example UTF-8). Use the encoding that matches your files to avoid mojibake or parse failures. |
Max file size (bytes) | Upper bound on file size for ingestion. 0 often means no limit or use product default—confirm in your build. Non-zero values reject or skip oversized files early. |
Field | Description |
|---|---|
Splitter type | How text is split into chunks before embedding. Recursive (when labeled recommended) usually splits on paragraphs and headings first, then sentences, for more coherent chunks. Other types (if offered) may split on fixed characters or delimiters only. |
Chunk size (characters) | Target maximum size of each chunk in characters (not tokens). Larger chunks preserve more context but can reduce retrieval precision; smaller chunks improve granularity but increase vector count and cost. Default 1000 is a common starting point. |
Chunk overlap (characters) | Number of characters shared between adjacent chunks. Overlap helps avoid cutting sentences or facts in half at boundaries. Default 200 is typical with 1000-character chunks; adjust if answers miss context at edges. |
Custom separators (optional) | Extra delimiter strings (if your product supports them) that force splits—for example specific headings or markers. Leave empty to use the splitter’s built-in rules. |
Field | Description |
|---|---|
Batch size | How many chunks or documents to process per internal batch (for example 100). Higher values can improve throughput but increase memory use and failure blast radius on errors. |
Continue on error | When enabled, ingestion skips or logs failed files or chunks and continues with the rest. When disabled, the job may stop on the first error—better for strict validation; worse for large mixed folders. |
Control | Description |
|---|---|
Run ingestion | Starts the ingestion job with the current settings. Ensure vector store and paths are correct before running; large directories can take a long time. |
Show / hide advanced settings | Toggles visibility of parser, chunking, and ingestion options. Basic runs may only need source + vector store. |