DDQ Automation Platform | Grayscale Investments

Overview

The DDQ (Due Diligence Questionnaire) Automation Platform is an AI-powered system I built at Grayscale to automate what was previously a fully manual process for the Operations team.

Institutional investors and partners regularly send due diligence questionnaires as part of their evaluation process. These questionnaires often contain hundreds of questions, many of which are similar across submissions. Before this platform, the Ops team had to manually search through source documents, verify accuracy, and draft responses each time. It was repetitive, time-consuming, and prone to inconsistency.

I embedded directly with the Ops team to understand their workflow, then rebuilt it as an agentic system that compresses a multi-hour manual process into something significantly faster and more consistent.

Why Not Standard RAG?

Early on, I evaluated off-the-shelf solutions (AutoRFP.ai, Loopio, DiligenceVault, SiftHub) and standard vector RAG approaches. The core issues with pure vector RAG for DDQ answering:

• Chunking destroys answer quality. Vector ingestion chops complete, human-reviewed answers into ~300-token fragments. The retriever may only surface half of a complete answer.

• DDQ answers require synthesis, not retrieval. A question like "Describe your approach to third-party risk management" may require information from vendor assessments, SOC 2 controls, procurement policies, and integration security reviews. Vector search retrieves independently similar chunks but cannot reason about how they relate.

• Semantic similarity fails on DDQs. Questions like "How do you handle data deletion?" and "How do you handle data retention?" have near-identical embeddings but completely different correct responses.

Building in-house allowed us to implement a more sophisticated approach.

The Wiki-Primary Architecture

Inspired by Andrej Karpathy's LLM Knowledge Base pattern, the platform uses a wiki-primary architecture where an LLM compiles and maintains a structured knowledge base from raw sources. At query time, the LLM reads from pre-synthesized, organized content rather than raw fragments.

The wiki lives as a directory of Markdown files in S3, mounted as a native filesystem via Amazon S3 Files. Every compute surface accesses the same wiki through standard file I/O.

Advantages:

• Pre-synthesized knowledge. The compilation step deduplicates, resolves contradictions, and organizes by concept. At query time, the LLM reads coherent, pre-organized material rather than a bag of chunks.

• Full answers preserved. Complete, polished answers from the master Q&A database are maintained as whole articles, enriched with context from policy docs and product specs.

• Reasoning-based retrieval. The LLM reads a compact index of all articles and reasons about which are relevant, rather than relying on embedding similarity.

• Natural citations. Each wiki article tracks provenance back to its source material.

• Cost-efficient at runtime. The two-step retrieval pattern (scan compact index → load only relevant articles) yields roughly 10x cost savings over raw ingestion approaches.

A parallel Bedrock Knowledge Base over the raw source documents provides a verification layer, ensuring answer quality even if the wiki compilation introduces errors.

Technical Architecture

The system is hosted entirely on AWS:

• ECS Fargate for containerized compute

• S3 for document storage (wiki mounted as native filesystem)

• Bedrock Knowledge Base for the verification layer

• Claude Sonnet 4.6 (1M token context) for answer generation

• Claude Haiku 4.5 for query classification and routing

• TypeScript MCP server with Streamable HTTP transport, integrating directly with the Claude.ai web interface

• OAuth 2.1 via Microsoft Entra ID for user authentication

The DDQ processing pipeline is a deterministic TypeScript outer loop that iterates questions by category and calls Claude for answer generation. An adaptive routing system sends each question through the appropriate retrieval path: lightweight lookup for simple questions, full wiki index scan for complex synthesis.

User corrections to generated answers feed back into the knowledge base, creating a compounding quality loop where the system improves with every DDQ it processes.

My Role

I owned requirements definition for the platform, working closely with our lead engineer on architecture planning and ideation. I also facilitated testing with business stakeholders from the Operations team and continue to optimize data structures to improve model outputs.

This project also served as an internal proof point for AI adoption at Grayscale, demonstrating how agentic systems can automate real operational work.

Current State

The platform is live and being continuously refined. Operations has given glowing feedback on the outputs so far. I'm currently testing with stakeholders and optimizing data structures to improve retrieval accuracy and response quality.

Additional Thoughts

I'm proud of the architectural decisions on this project. It would have been easy to reach for standard vector RAG or buy an off-the-shelf solution, but taking the time to understand the problem deeply led us to a more sophisticated approach that genuinely outperforms those alternatives.

The wiki-primary pattern is something I'd use again. It front-loads the hard work (deduplication, synthesis, organization) into the compilation step, where it can be done once, done well, and validated. At query time, the LLM is working with clean, pre-organized material rather than trying to make sense of raw fragments on the fly.