Skip to content

WIP: RAG chat cookbook with interstitial text#23

Draft
djliden wants to merge 5 commits intomainfrom
rag-chat-template
Draft

WIP: RAG chat cookbook with interstitial text#23
djliden wants to merge 5 commits intomainfrom
rag-chat-template

Conversation

@djliden
Copy link
Copy Markdown
Collaborator

@djliden djliden commented Apr 6, 2026

Summary

  • Replaces scaffold-rag-chat with rag-chat-integration capstone recipe containing full RAG app code
  • Adds rag-chat-app-template.tsx page component (fixes broken template link)
  • Adds connecting text between recipe phases in the cookbook view (architecture overview, phase transitions)
  • Fixes @ai-sdk/openai@3 version pin and improves several recipe code blocks
  • Enhances lakebase-chat-persistence with setupChatTables, listChats, getChatMessages

Status: WIP - Cookbook content and template page are functional. Agent one-shot build verified via dv2 dispatch (cursor-coding agent successfully built working RAG chat app from cookbook markdown alone).

Deploy difficulties

Deployment to Databricks Apps is currently blocked:

  • package-lock.json generated locally resolves all tarballs to npm-proxy.dev.databricks.com
  • Databricks App containers cannot reach this proxy, causing npm install to fail with ETIMEDOUT on throttleit-2.1.0.tgz
  • Workaround: exclude package-lock.json from deploy bundle + ship .npmrc pointing to registry.npmjs.org — not yet verified end-to-end
  • Alternative: ship pre-built artifacts (dist/, client/dist/, production node_modules/) so container skips install entirely

Test plan

  • Verify template page renders at /resources/rag-chat-app-template with all 8 recipes + connecting text
  • Verify npm run build succeeds (no broken links)
  • End-to-end: deploy cookbook-built app to Databricks and confirm RAG chat works
  • Verify bun run test smoke tests pass

This pull request was AI-assisted by Isaac.

djliden added 3 commits April 2, 2026 13:09
Introduces a typed block system (markdown, recipe, code) that lets
templates interleave prose and code between recipe steps instead of
just concatenating recipes. Refactors all template pages and the
markdown export API to use the new system, with a fallback to legacy
concatenation for templates without custom blocks.

Co-authored-by: Isaac
Add three new recipes for RAG chat application development:
- embeddings-generation.md: Guide for generating embeddings from documents
- lakebase-pgvector.md: PostgreSQL vector storage with pgvector extension
- scaffold-rag-chat.md: Scaffolding a RAG chat application

Update recipe registry (recipes.ts) to include new RAG chat recipes with proper metadata, prerequisites, and categorization. Update content-entries plugin to support new recipe structure. Fix tsconfig path mapping for improved type resolution.

Co-authored-by: Isaac
- Add rag-chat-integration.md: complete capstone recipe covering document
  seeding, RAG-augmented chat route, sources API, chat persistence routes,
  server bootstrap, client sources display, and App.tsx root route setup
- Delete scaffold-rag-chat.md (was just CLI scaffolding with no RAG logic)
- Create rag-chat-app-template.tsx page component (fixes broken link)
- Update recipes.ts registry (swap slug, update recipeIds)
- Enhance template-content.ts with connecting text between recipe phases
- Fix @ai-sdk/openai@3 version pin in ai-chat-model-serving recipe
- Improve embeddings-generation, lakebase-pgvector, lakebase-chat-persistence
  recipes with fuller code blocks and setupChatTables/listChats/getChatMessages

Verified: an agent can one-shot build a working RAG chat app from the
cookbook markdown alone (tested via dv2 dispatch). Deploy to Databricks
Apps blocked by npm proxy issues (package-lock.json resolved URLs point
to npm-proxy.dev.databricks.com which is unreachable from app containers).

Co-authored-by: Isaac
"rag-chat-app-template": [
{
type: "markdown",
content: `## What you are building
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we move this into content/cookbooks/ or similar and instead link to the markdown file here? Much easier to author and manage markdown in .md files.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes working on that now

tsconfig.json Outdated
}
},
"exclude": [".docusaurus", "build", "tests"]
"exclude": [".docusaurus", "build", "tests", "templates"]
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't look like templates/ is part of this PR? 👀

djliden added 2 commits April 6, 2026 16:38
No templates/ directory exists in the repo.

Co-authored-by: Isaac
Extend the content-entries plugin to support a data-only mode (no routes)
for loading cookbook markdown at build time, eliminating the fragile
raw-loader webpack hack.

Co-authored-by: Isaac
@djliden
Copy link
Copy Markdown
Collaborator Author

djliden commented Apr 7, 2026

Updated.

To add a cookbook comprising recipes and connecting text, you now:

  1. Create a markdown file in content/cookbooks/your-cookbook.md
  2. Write narrative text as normal markdown, embedding recipes with <!-- recipe: recipe-id --> markers where they should appear
  3. The content-entries plugin picks up the file automatically at build time; no TypeScript content to maintain

return markdown.trim();
}

export function buildLegacyTemplateRawMarkdown(
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would just move all templates to the new logic in this PR instead of creating two paths (legacy, new)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants