Skip to content

Commit a36d7e1

Browse files
docs: document FTS block_size index option (#322)
Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com>
1 parent c502983 commit a36d7e1

1 file changed

Lines changed: 19 additions & 0 deletions

File tree

docs/indexing/fts-index.mdx

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,7 @@ await async_table.create_index("payload.text", config=FTS(with_position=True))
9696
| `ngram_min_length` | int | `3` | Minimum n-gram length. Applies only when `base_tokenizer="ngram"`. |
9797
| `ngram_max_length` | int | `3` | Maximum n-gram length. Applies only when `base_tokenizer="ngram"`. |
9898
| `prefix_only` | bool | `False` | Index only prefix n-grams rather than all substrings. Applies only when `base_tokenizer="ngram"`. |
99+
| `block_size` | int | `128` | Number of documents per compressed posting block. Supported values are `128` and `256`. Setting this to `256` opts in to the experimental FTS V3 layout. |
99100

100101
<Note title="Key parameters">
101102
- `max_token_length` can filter out base64 blobs or long URLs.
@@ -117,6 +118,24 @@ Model-backed tokenizers such as `jieba/default`, `lindera/ipadic`, and `lindera/
117118

118119
`language` is used by token filters, not by the base tokenizer. Stemming supports Arabic, Danish, Dutch, English, Finnish, French, German, Greek, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish, Tamil, and Turkish. Built-in stop-word removal supports Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Russian, Spanish, and Swedish. For other stemming languages, set `remove_stop_words=False` or pass `custom_stop_words`.
119120

121+
### Posting block size
122+
123+
`block_size` controls the number of documents packed into each compressed posting block on disk. The default of `128` matches the current FTS layout and is the right choice for most workloads. Setting it to `256` opts in to the experimental FTS V3 format, which changes how postings are encoded and may introduce breaking changes in future releases. Any other value is rejected at index creation time.
124+
125+
You can set the option through either the synchronous or asynchronous API. In the async Python API, pass it on the `FTS` config, and in the TypeScript API use the camelCase `blockSize` field on `FtsOptions`:
126+
127+
```python Python icon="python"
128+
from lancedb.index import FTS
129+
130+
await async_table.create_index("text", config=FTS(block_size=256))
131+
```
132+
133+
```typescript TypeScript icon="square-js"
134+
await table.createIndex("text", {
135+
config: lancedb.Index.fts({ blockSize: 256 }),
136+
});
137+
```
138+
120139
### Phrase Query Configuration
121140

122141
Enable phrase queries by setting:

0 commit comments

Comments
 (0)