AI
From Big Data to Heavy Data: Rethinking the AI Stack
LLMs can finally interpret unstructured video, audio, and documents — but they can't do it alone. This post introduces the concept of heavy data and explores how modern teams build multimodal pipelines to turn it into AI-ready data.
Scalable PDF Document Processing with DataChain and Unstructured.io
Extract and parse text from documents and create vector embeddings in a scalable and distributed way (and less than 70 lines of code).
Enforcing JSON Outputs in Commercial LLMs
The results of our tests on the structured outputs of Google Gemini Pro, Anthropic Claude, and OpenAI GPT. DataChain used for evaluation.
Dataset Factory - A Toolchain for Generative Computer Vision Datasets
Learn about our latest approach to mastering your Unstructured Data and metadata.
