Plugin Author
rapls
(@rapls)
@publiastel
Thanks for the detailed report, and for testing with your own OpenAI key. You have found a real gap, so let me be straight about what is happening.
On the size limit, there are two of them and they work differently. The upload itself is capped at 5 MB and accepts txt, csv, md, pdf, and docx. But the one that is biting you is not a file-size limit at all. Uploaded documents are stored and embedded whole; the plugin does not split them into chunks (the chunking you may have read about is done by the site crawler, not by document uploads). So each document has to fit inside the embedding model’s single-request token limit. For OpenAI text-embedding-3-small that is about 8192 tokens, roughly 30,000 characters. Go over that and the provider rejects the request, the plugin skips the document, and you get the silent “0 chunks” you saw. A scanned or encrypted PDF is a different cause: the parser cannot pull any text out of it, and that case does show an error at upload time.
On whether Pro changes this: no. Pro runs the same embedding pipeline, so the per-document token limit and the whole-document behavior are identical in both. What Pro adds on the RAG side is visibility, an embedded-versus-total progress indicator, plus some context and embedding controls. It does not raise the limit. I would rather say that plainly than have you buy Pro expecting a different result.
Your third point is the one I most agree with. Indexing 0 and saying nothing is bad behavior, and I am going to fix it: a document that fails to embed, whether it is too large or has no extractable text, should show the reason on the Knowledge Base screen instead of just vanishing. Thank you for pushing on it.
Two things that will get you working today:
– Split large documents into smaller files before uploading, so each part stays well under the model’s limit. A few thousand words per file is a safe target, and each part will embed on its own.
– After uploading, run Generate Embeddings and compare the embedded count against the total. Anything that stays unembedded is almost certainly an oversized file. Turn on WP_DEBUG and the exact provider error lands in the log if you want to confirm it.
Thanks again for the clear write-up. Reports that pin down the reproduction like this are the ones that get fixed.
Plugin Author
rapls
(@rapls)
@publiastel
This is fixed in 1.18.1, which I just released.
Two things changed from what you hit:
– A document that fails to embed no longer indexes silently. The Knowledge Base list now shows the reason on the entry itself. It says “too large to embed”, or gives an API-key / rate-limit note, instead of the 0-chunk silence you ran into.
– If you upload several documents and one is over the model’s token limit, that one no longer takes the whole batch down with it. The rest embed normally and only the oversized one gets flagged. There was also a loop that could keep re-sending embedding requests when a document could never succeed; that is fixed too, so it stops instead of running on.
On the part you would actually want next, the plugin splitting a large document into chunks for you so it just works without pre-splitting: I am treating that as its own separate release. It is a bigger change to the ingestion side than the fix above, and I would rather ship it on its own than bolt it onto a bug-fix. It is on the list.
Until then the workaround is the same: split a large file into smaller entries (a few thousand words each) before uploading, and each part embeds on its own. After uploading, run Generate Embeddings and check the embedded count against the total; anything left over now tells you why on the list.
Thanks again for the report. Pinning down the reproduction the way you did is what got this fixed quickly.
