Bulk Document Ingestion

Back to Documents Single Document

Bulk Ingestion Form

Input Type

URLs (One per line)

File Uploads

Mixed (URLs + Files)

URLs (One per line) *

Enter one URL per line. Supports webpages, PDFs, and other documents accessible via URL.

Content Type

Select the content type for all documents in this batch.

Deduplication

Skip existing items

When enabled, URLs and files that have already been ingested will be skipped. This prevents duplicate processing and saves time/API calls.

Help & Tips

Supported Input Types:

URLs: Webpages, PDFs, images accessible via URL
Files: PDF, DOCX, DOC, TXT, RTF, images, audio, video
Mixed: Combine URLs and file uploads

Rate Limiting:

• OpenAI: 3 requests/second (333ms delay)
• Gemini: 15 requests/second (67ms delay)
• Default: 1 request/second (1000ms delay)

Best Practices:

• Start with a small batch (5-10 items)
• Use longer delays for large batches
• Check URLs are publicly accessible
• Monitor for rate limit errors
• Large files may take longer to process