I saw Pietro Schirano's tweet about his "chain of tools" and immediately thought - I must build this! You see, I've been obsessed with Claude's tool use capabilities lately, especially after Anthropic released their interleaved thinking features. Most Claude integrations I'd seen treated tools as black boxes - call a tool, get a result, move on. But what if tool results could feed back into Claude's thinking process in real-time?

That simple question led me down a rabbit hole that resulted in ThinkChain - a system where thinking, tool execution, and reasoning form a continuous feedback loop. Instead of the traditional linear flow of "call tool → get result → respond," ThinkChain creates something much more powerful: "think → call tool → think about results → respond intelligently."

What I discovered surprised me. When you inject tool results back into Claude's thinking stream, it doesn't just use tools - it becomes dramatically smarter about how it uses them. Here's what I built, what I learned, and why this changes everything about AI tool integration.

The Core Innovation: Tool Result Injection

Let me show you the difference with a real example. Ask a traditional Claude integration "What's the weather in San Francisco and where should I eat dinner there?" and you get this flow:

Traditional approach:

User Question → Claude thinks → Calls weather tool → Gets result
               → Calls restaurant tool → Gets result → Combines results

ThinkChain approach:

User Question → Claude thinks → Calls weather tool → Thinks about weather
               → Calls restaurant tool with weather context → Thinks about both
               → Synthesizes intelligent response

The magic happens in those thinking steps after tool execution. Here's the actual technical implementation that makes this possible:

async def stream_once(messages, tools):
    async with client.messages.stream(
        model="claude-sonnet-4-20250514",
        messages=messages,
        tools=tools,
        extra_headers={"anthropic-beta": "interleaved-thinking-2025-05-14,fine-grained-tool-streaming-2025-05-14"},
        thinking={"type": "enabled", "budget_tokens": 1024}
    ) as stream:

        async for event in stream:
            if event.type == "tool_use":
                # Execute the tool
                result = await execute_tool(event.name, event.input)

                # This is the key: inject result back into thinking stream
                transcript.append({
                    "role": "user",
                    "content": [{"type": "tool_result", "content": result}]
                })

                # Continue streaming - Claude thinks about the results
                return await stream_once(transcript)

This creates a feedback loop where Claude's initial thinking leads to tool use, tool results inform continued thinking, and the final response incorporates both reasoning and tool outcomes. It's not just smarter - it's thinking smarter.

Architecture Deep Dive: How It All Works

Building ThinkChain taught me that the real power isn't in having lots of tools - it's in how tools discover each other, execute cleanly, and feed results back intelligently. Here's how I architected it:

The Tool Discovery System

I wanted developers to just drop a Python file in a folder and have it work. No registration, no complex setup. Here's the discovery pipeline:

Local Tools (/tools/*.py) → Validation → Registry
                                         ↓
MCP Servers (config.json) → Connection → Registry → Unified Tool List → Claude API

Every tool implements this simple interface:

from tools.base import BaseTool

class WeatherTool(BaseTool):
    name = "weathertool"
    description = """
    Gets current weather information for any location worldwide.

    Use this tool when users ask about:
    - Current weather in any city/location
    - Temperature anywhere
    - "What's the weather like in [location]?"
    """

    input_schema = {
        "type": "object",
        "properties": {
            "location": {"type": "string", "description": "City and state/country"},
            "units": {"type": "string", "enum": ["fahrenheit", "celsius"], "default": "fahrenheit"}
        },
        "required": ["location"]
    }

    def execute(self, **kwargs) -> str:
        location = kwargs.get("location")
        # Hit wttr.in API for real weather data
        response = requests.get(f"https://wttr.in/{location}?format=j1")
        data = response.json()

        # Format for Claude
        return f"🌤️ Weather for {location}:\nTemperature: {data['current_condition'][0]['temp_F']}°F\n..."

The beauty is that tools are just classes with four things: a name, description, input schema, and execute method. Drop the file in /tools/, and ThinkChain discovers it automatically.

Real Example Flow

Let me show you what happens when someone asks "What's the weather in San Francisco and find good restaurants there?":

[thinking] I need to check the weather first, then find restaurants that might be good for those conditions.

[tool_use:weathertool] ▶ {"location": "San Francisco, CA"}
[tool_result] 🌤️ Weather for San Francisco, CA:
Temperature: 62°F (feels like 62°F)
Conditions: Partly cloudy
Humidity: 38%
Wind: 5 mph WSW

[thinking] It's a pleasant 62°F and partly cloudy - perfect weather for outdoor dining or walking to restaurants. I should look for places with outdoor seating or patios.

[tool_use:duckduckgotool] ▶ {"query": "best restaurants San Francisco outdoor seating patio"}
[tool_result] [Restaurant results with outdoor dining options...]

[thinking] Given the nice weather, I can recommend these outdoor-friendly restaurants...

See how the weather result influences the restaurant search? That's the power of tool result injection - Claude doesn't just call tools sequentially, it thinks about results and makes smarter decisions.

Building Real Tools: From Concept to Code

When I started building tools for ThinkChain, I learned that the description is just as important as the implementation. Claude needs to understand not just what your tool does, but when to use it.

Here's the complete weathertool implementation with everything I learned:

from tools.base import BaseTool
import requests
import json

class WeatherTool(BaseTool):
    name = "weathertool"

    # This description is crucial - it helps Claude decide when to use the tool
    description = """
    Gets current weather information for any location worldwide. Returns temperature, 
    weather conditions, humidity, wind speed and direction.

    Use this tool when users ask about:
    - Current weather in any city/location
    - Temperature anywhere  
    - Weather conditions (sunny, cloudy, rainy, etc.)
    - "What's the weather like in [location]?"
    """

    # JSON Schema for input validation
    input_schema = {
        "type": "object",
        "properties": {
            "location": {
                "type": "string",
                "description": "The city and state/country (e.g., 'San Francisco, CA' or 'London, UK')"
            },
            "units": {
                "type": "string", 
                "description": "Temperature units",
                "enum": ["fahrenheit", "celsius", "kelvin"],
                "default": "fahrenheit"
            }
        },
        "required": ["location"]
    }

    def execute(self, **kwargs) -> str:
        location = kwargs.get("location")
        units = kwargs.get("units", "fahrenheit")

        try:
            # Use wttr.in - free weather API, no key needed
            response = requests.get(f"https://wttr.in/{location}?format=j1", timeout=10)
            response.raise_for_status()
            data = response.json()

            current = data['current_condition'][0]
            temp_c = int(current['temp_C'])
            temp_f = int(current['temp_F'])

            # Format based on requested units
            if units.lower() == "celsius":
                temp = f"{temp_c}°C"
            else:  # Default to fahrenheit
                temp = f"{temp_f}°F"

            # Return formatted result that Claude can easily understand
            return f"""🌤️ Weather for {location}:
Temperature: {temp}
Conditions: {current['weatherDesc'][0]['value']}
Humidity: {current['humidity']}%
Wind: {current['windspeedMiles']} mph {current['winddir16Point']}"""

        except Exception as e:
            # Always return string errors - Claude can handle them gracefully
            return f"❌ Error fetching weather data: {str(e)}"

Key Patterns I Discovered

Rich Descriptions Win: The more context you give Claude about when to use your tool, the better it performs. Include example queries, keywords that should trigger it, and specific use cases.

Error Handling Matters: Always catch exceptions and return string error messages. Claude is surprisingly good at handling errors gracefully when you give it clear information about what went wrong.

Format for Claude: Structure your output to be easily parseable. Use emojis, clear labels, and consistent formatting. Claude works better with well-structured data.

Input Validation: Use comprehensive JSON schemas. They prevent errors and help Claude understand exactly what parameters your tool expects.

MCP Integration: Extending Beyond Local Tools

One of the most exciting discoveries was integrating with MCP (Model Context Protocol) servers. MCP lets you connect to external servers that provide tools, dramatically expanding what's possible.

Here's how I added SQLite database operations:

{
  "mcpServers": {
    "sqlite": {
      "command": "uvx",
      "args": ["mcp-server-sqlite", "--db-path", "./database.db"],
      "description": "SQLite database operations",
      "enabled": true
    }
  }
}

Just by adding that configuration and running uvx install mcp-server-sqlite, ThinkChain gained 6 new tools:

• mcp_sqlite_read_query - Execute SELECT queries

• mcp_sqlite_write_query - Execute INSERT/UPDATE/DELETE

• mcp_sqlite_create_table - Create database tables

• mcp_sqlite_list_tables - List all tables

• mcp_sqlite_describe_table - Get table schema

• mcp_sqlite_append_insight - Add business insights

The power comes from combining ecosystems. Now I can ask: "Check the weather in our office locations from the database, then find restaurants near each one" and Claude seamlessly uses both local tools (weather) and MCP tools (database) together.

What blew my mind was how naturally Claude chains these together. It doesn't see a difference between local Python tools and remote MCP servers - they're all just tools in its toolkit.

The Enhanced UI: Making It Beautiful

Here's something I learned early: if you're building developer tools, the experience matters just as much as the functionality. I could have stopped at a basic CLI, but I wanted ThinkChain to feel as intelligent as it actually is.

So I built two interfaces:

thinkchain.py - The full experience with Rich formatting, progress bars, and interactive features thinkchain_cli.py - Minimal CLI for when you just need it to work run.py - Smart launcher that detects available libraries and picks the best option

Here's what the enhanced UI looks like when it starts up:

╔═══════════════════════════════════════════════════════════════════╗
║  ████████╗██╗  ██╗██╗███╗   ██╗██╗  ██╗ ██████╗██╗  ██╗ █████╗ ██╗███╗   ██╗  ║
║  ╚══██╔══╝██║  ██║██║████╗  ██║██║ ██╔╝██╔════╝██║  ██║██╔══██╗██║████╗  ██║  ║
║     ██║   ███████║██║██╔██╗ ██║█████╔╝ ██║     ███████║███████║██║██╔██╗ ██║  ║
║     ██║   ██╔══██║██║██║╚██╗██║██╔═██╗ ██║     ██╔══██║██╔══██║██║██║╚██╗██║  ║
║     ██║   ██║  ██║██║██║ ╚████║██║  ██╗╚██████╗██║  ██║██║  ██║██║██║ ╚████║  ║
║     ╚═╝   ╚═╝  ╚═╝╚═╝╚═╝  ╚═══╝╚═╝  ╚═╝ ╚═════╝╚═╝  ╚═╝╚═╝  ╚═╝╚═╝╚═╝  ╚═══╝  ║
║          🧠 Claude Chat with Advanced Tool Integration & Thinking 💭          ║
╚═══════════════════════════════════════════════════════════════════════════════╝

Claude Tool Discovery Chat
🔧 Local: 11 tools │ 🌐 MCP: 6 servers │ 💭 Thinking: ON │ 🔋 Ready

But the real magic happens during conversations. Watch what happens when Claude uses a tool:

👤 You: What's the weather in Cross Lanes, WV?

💭 Thinking: I'll check the current weather in Cross Lanes, WV for you.

🔧 Tool Use: weathertool

🔧 Executing: weathertool
╭───────────────────────── Arguments for weathertool ──────────────────────────╮
│ {                                                                            │
│   "location": "Cross Lanes, WV"                                              │
│ }                                                                            │
╰──────────────────────────────────────────────────────────────────────────────╯
🔧 weathertool: Executing...
🔧 weathertool: Completed (0.8s)
╭────────────────────────── Result from weathertool ───────────────────────────╮
│ 🌤️ Weather for Cross Lanes, WV:                                               │
│ Temperature: 73°F (feels like 77°F)                                          │
│ Conditions: Heavy rain with thunderstorm                                     │
│ Humidity: 79%                                                                │
│ Wind: 2 mph WSW                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯

🔄 Continuing with tool results...

💭 Thinking: The current weather in Cross Lanes, WV shows stormy conditions...

🤖 Claude: The current weather in Cross Lanes, WV is stormy with heavy rain and thunderstorms...

Every step is visualized: thinking appears in italic blue, tool execution shows progress with timing, and results are formatted in beautiful boxes. You can actually watch Claude think through problems.

Technical Implementation

The enhanced UI is built with the Rich library, but here's the clever part - it gracefully degrades:

try:
    from rich.console import Console
    from rich.panel import Panel
    from rich.progress import Progress
    from ui_components import ui  # Enhanced UI components
    UI_AVAILABLE = True
except ImportError:
    UI_AVAILABLE = False

def print_tool_execution(name, status, duration=None):
    if UI_AVAILABLE:
        if status == "executing":
            ui.print(f"🔧 [yellow]Executing:[/yellow] {name}")
        elif status == "completed":
            ui.print(f"🔧 [green]Completed:[/green] {name} ({duration:.1f}s)")
    else:
        # Fallback to basic text
        print(f"[tool_use:{name}] {status}")

The slash commands system was particularly fun to build:

command_words = ['/help', '/tools', '/refresh', '/config', '/exit']

user_input = ui.get_input("Enter command or message", command_words)

if user_input.startswith('/'):
    command_parts = user_input[1:].split()
    command = command_parts[0].lower()

    if command == 'tools':
        show_tools_command()  # Beautiful table of all tools
    elif command == 'config':
        handle_config_command(args)  # Interactive configuration

You get tab completion, command history with arrow keys, and rich formatting throughout. But if you don't have Rich installed, everything still works - it just falls back to plain text.

Lessons Learned and Developer Insights

Building ThinkChain taught me things about AI tool integration that I never expected. Here are the biggest insights:

What Worked Incredibly Well

Tool result injection is a game-changer. I cannot overstate this. When Claude can think about tool results before responding, the quality of responses improves dramatically. It's not just using tools - it's reasoning about their outputs.

Automatic tool discovery scales effortlessly. I started with 2 tools, now have 17, and adding new ones is still just "drop file in folder, restart." The discovery system handles all the complexity.

Rich descriptions make Claude smarter. The difference between a tool with a basic description and one with rich context about when to use it is night and day. Claude makes much better tool selection decisions with good descriptions.

MCP integration unlocks unlimited possibilities. Once I connected to MCP servers, I realized this isn't just about the tools I build - it's about connecting to an entire ecosystem.

Challenges That Surprised Me

Managing async MCP connections was trickier than expected. MCP servers run as separate processes, and coordinating their lifecycle with the main application required careful async handling:

async def cleanup_mcp_servers():
    """Gracefully shutdown all MCP server connections"""
    for server_name, client in self.active_clients.items():
        try:
            await client.close()
        except Exception as e:
            logger.error(f"Error during cleanup of MCP server {server_name}: {e}")

Tool failure handling needs to be bulletproof. When a tool fails, you can't just crash - Claude needs to understand what went wrong and potentially try alternative approaches:

def execute_tool_sync(name: str, args: dict) -> str:
    try:
        result = tool_function(args)
        return result
    except requests.RequestException as e:
        return f"❌ Network error calling {name}: {str(e)}"
    except ValidationError as e:
        return f"❌ Invalid input for {name}: {str(e)}"
    except Exception as e:
        return f"❌ Unexpected error in {name}: {str(e)}"

Thinking budget optimization matters more than I thought. Initially I set the thinking budget to 16,000 tokens, but I found that 1,024-2,048 tokens often works better. Too much thinking budget and Claude overthinks simple problems. Too little and it can't reason through complex tool chains.

Performance Insights

Tool execution timing varies wildly. Weather API calls take 0.5-1 second, web scraping can take 3-5 seconds, and database operations are nearly instant. The UI progress indicators help users understand what's happening.

Streaming vs. batch processing trade-offs. Streaming gives better user experience but requires more complex error handling. I ended up with a hybrid approach - streaming for the conversation, but batch processing for tool discovery at startup.

Memory usage grows with tool count. Each tool keeps its schema in memory, and MCP connections maintain persistent state. With 17 tools I'm using about 50MB, which is totally reasonable, but it's something to watch.

Code Patterns That Emerged

Here are some patterns I found myself using repeatedly:

# Tool result validation pattern
def validate_and_format_result(result: str, tool_name: str) -> str:
    if not result:
        return f"❌ {tool_name} returned empty result"

    # Try to parse as JSON for structured data
    try:
        parsed = json.loads(result)
        return json.dumps(parsed, indent=2)  # Pretty print
    except:
        return result  # Return as-is if not JSON

# Graceful degradation pattern  
def safe_tool_execution(tool_func, *args, **kwargs):
    try:
        return tool_func(*args, **kwargs)
    except ImportError as e:
        return f"❌ Missing dependency: {e}"
    except Exception as e:
        return f"❌ Tool execution failed: {e}"

# Configuration management pattern
def update_config(key: str, value: Any) -> bool:
    if key in ALLOWED_CONFIG_KEYS:
        CONFIG[key] = value
        save_config_to_file()  # Persist changes
        return True
    return False

What I realized is that building AI tools isn't just about the AI part - it's about creating robust, developer-friendly systems that handle edge cases gracefully and provide great experiences.

Fork It and Make It Yours

Here's the thing - ThinkChain is designed to be forked and extended. I built it with MIT license specifically because I want to see what developers build with it.

The architecture is modular by design. Want to add tools for your domain? Drop Python files in /tools/. Want to connect to specialized MCP servers? Edit mcp_config.json. Want to customize the UI? Modify the Rich components.

Ideas for Domain-Specific Forks

Data Science ThinkChain: Add pandas tools for data manipulation, matplotlib for visualization, jupyter tools for notebook integration. Imagine asking Claude to "load this dataset, analyze trends, and create visualizations" and watching it think through each step.

Web Development ThinkChain: React component generators, npm package managers, git integration tools, deployment automation. "Create a new React component with these props and add it to the project" becomes a conversation, not a manual process.

DevOps ThinkChain: Docker container tools, Kubernetes deployment tools, AWS/GCP integration, monitoring dashboards. "Check the health of our production services and scale if needed" with full reasoning about the decisions.

Research ThinkChain: Academic paper search tools, citation managers, data analysis tools, LaTeX generators. "Find recent papers on this topic and summarize their methodologies" with tool-driven research.

Getting Started with Your Fork

The process is straightforward:

# Fork and clone
git clone https://github.com/yourusername/your-thinkchain-fork.git
cd your-thinkchain-fork

# Install dependencies
uv pip install -r requirements.txt

# Create your first tool
vim tools/yourtool.py

# Test it
python thinkchain.py
/refresh  # Loads your new tool
"Use my new tool for X"  # Test with Claude

What I Hope You Build

I'm excited to see domain-specific forks, novel tool combinations, and creative MCP integrations. Maybe someone builds ThinkChain for legal research, or scientific computing, or creative writing. The possibilities are endless.

If you build something cool, let me know! I'd love to feature community forks and see how people extend the system.

What's Next

Building ThinkChain opened my eyes to what's possible when AI tools can think about their own tool use. Here's what I'm excited about for the future:

Technical Improvements I'm Working On

Better error recovery: When tools fail, Claude should be able to suggest alternative approaches or debug the problem. I'm experimenting with giving Claude access to error logs and system state.

Tool composition workflows: Instead of just chaining tools, what if Claude could compose them into reusable workflows? "Remember this sequence of tools as a 'data analysis workflow' for future use."

Multi-model support: Claude is amazing, but different models have different strengths. What if you could use GPT-4 for creative tasks and Claude for analytical ones, all in the same conversation?

Performance optimizations: Some tool chains could run in parallel instead of sequentially. I'm exploring how to let Claude mark which tools can run concurrently.

The Bigger Picture

What excites me most is that ThinkChain represents a shift from "AI that uses tools" to "AI that thinks about tools." When Claude can reason about tool results, it makes fundamentally better decisions about which tools to use and how to use them.

I think this is just the beginning. As more MCP servers come online, as tool ecosystems mature, and as AI models get better at reasoning, we're going to see AI systems that don't just automate tasks - they intelligently orchestrate complex workflows.

The future isn't AI replacing human developers - it's AI becoming incredibly sophisticated development partners that can think through problems, use tools intelligently, and explain their reasoning every step of the way.

Conclusion

Pietro's tweet about "chain of tools" sparked an idea, but what I discovered while building ThinkChain was something bigger: when you let AI think about tool results, everything changes.

Claude doesn't just use tools anymore - it reasons about them, learns from them, and makes intelligent decisions about how to combine them. The feedback loop between thinking and tool execution creates a kind of intelligence I hadn't seen before.

For developers, this means we need to think differently about AI integration. It's not enough to just give AI access to tools - we need to design systems that let AI think about tool results and use that thinking to make better decisions.

The technical patterns are surprisingly straightforward: tool result injection, async streaming, graceful error handling, and rich user experiences. But the implications are profound. We're moving from AI assistants that follow scripts to AI partners that can reason through complex problems.

ThinkChain is my exploration of this idea, but it's really just the beginning. The best AI tools aren't just smart - they're tools that make AI smarter.

Fork it, extend it, and build something amazing. I can't wait to see what you create.

Want to try ThinkChain? Check out the GitHub repository and start building your own tool-thinking feedback loops.

Have ideas or questions? Find me on Twitter/X - I'd love to hear what you're building!

Hey there, code enthusiasts! Martin Bowling here! 👋 Today I'm pulling back the curtain on DeepBuild, our AI-powered software wizard. Get ready for a fun journey through the codebase where I'll show you how everything ticks, why we made certain choices, and how you can join the party with your own contributions!

🤔 What Is DeepBuild?

Think of DeepBuild as your AI-powered coding companion that can spawn entire software projects from scratch! We've mixed together some awesome ingredients:

• ⚛️ Next.js (powering our slick front end and serverless API routes)

• 📝 TypeScript (because who doesn't love type safety?)

• 🎨 Tailwind CSS (for that pixel-perfect styling)

• 🪝 Custom React hooks and context providers (keeping our state game strong)

• 💾 IndexedDB (via a custom file manager) to store and manage project data in your browser

When you put it all together, users can:

1. Dream up a project idea

2. Watch as AI crafts a detailed project blueprint

3. Have a chat with the AI to refine the details

4. See complete files materialize like magic! ✨

🎯 The Core Concept

Here's the secret sauce:

1. 🗣️ You describe your dream project

2. 🤖 The AI cooks up some JSON containing:

   • 📋 A Project Brief (the grand plan)

   • ❓ Clarifying questions (getting those details just right)

   • 📁 A file structure (the blueprint)

3. 💬 You chat with the AI to refine everything

4. ⚡ DeepBuild transforms those ideas into real, working code!

Here's a peek at the prompts making the magic happen:

export const BRIEF_SYSTEM_PROMPT = `
  You are DeepBuild, an elite software engineer...
  ...
  Remember: Always wrap your response in <final_json> tags and ensure it's valid JSON.
`;

export const IMPLEMENTATION_SYSTEM_PROMPT = `
  You are DeepBuild, an elite software engineer...
  ...
  Remember: Always wrap your response in <final_json> tags and ensure it's valid JSON.
`;

🏗️ A Tour of the Code

1. 🎨 Root Layout & Global Styles

• File: app/layout.tsx

• File: app/globals.css

Our layout.tsx is where the magic begins! It sets up the main HTML structure and brings in our FileProvider context:

export default function RootLayout({ children }: { children: React.ReactNode }) {
  return (
    <html lang="en" suppressHydrationWarning>
      <body className={inter.className}>
        <FileProvider>{children}</FileProvider>
      </body>
    </html>
  );
}

We're all in on Tailwind for styling, with dark mode support baked right into globals.css 🌙

2. 📊 Main Dashboard

• File: app/page.tsx

The dashboard is your command center! Here's where you can:

• 🆕 Create shiny new projects

• 🗑️ Clean up old ones

• 📦 Export as .zip files

• ⚙️ Tweak your AI settings

export default function Dashboard() {
  // Hooks, state, event handlers here...
  // ...
  return (
    <div className="container py-8 px-12">
      {/* Header + new project button */}
      {/* List of projects as ProjectCard */}
      {/* Settings dialog */}
    </div>
  );
}

3. 🧩 Components Galore

Our components/ folder is where all the real action happens! Here's what we're working with:

• 🎴 ProjectCard.tsx: Shows off each project with its current status

• 🆕 NewProjectModal.tsx: A sleek dialog for birthing new projects

• 🖼️ ProjectView.tsx: The main stage where all the magic happens

• ✏️ Editor.tsx: Powered by @monaco-editor/react for that pro-level coding experience

Check out how we tie everything together in ProjectView.tsx:

<div className="flex-1 flex">
  <div className="w-64 border-r border-border bg-muted/30">
    <FileTree
      files={project.files}
      selectedFile={selectedFile}
      onFileSelect={setSelectedFile}
      onFileRefresh={handleFileRefresh}
    />
  </div>

  <div className="flex-1 border-r border-border">
    {selectedFile ? (
      <Editor
        file={selectedFile}
        language={getLanguageFromPath(selectedFile.path)}
      />
    ) : (
      <div className="h-full flex items-center justify-center text-muted-foreground">
        Select a file to view or edit
      </div>
    )}
  </div>

  <div className="w-96 flex flex-col h-full">
    <ProjectChat
      // ...
    />
  </div>
</div>

4. 💾 IndexedDB 'FileManager'

• File: lib/fileOperations.ts

We're keeping it local with IndexedDB for data storage! Our FileManager class handles all the heavy lifting:

class FileManager {
  private db: IDBDatabase | null = null;

  constructor() {
    this.initDB();
  }

  async saveProject(name: string, brief: ProjectBrief): Promise<string> {
    // Creates a new project + files in IndexedDB
  }

  // ...
}

No server required - everything stays right in your browser! 🏠

5. 🤖 Prompts and AI Hooks

• File: hooks/useChat.ts

The AI communication happens through sendChatMessage() in lib/api.ts. We can talk to either the DeepSeek or Hyperbolic model, depending on your preferences. We scan responses for those special <final_json> tags to extract the structured data we need:

const response = await sendChatMessage(apiMessages);
const match = response.match(/<final_json>([^]*?)<\/final_json>/);
let parsedResponse = null;

if (match) {
  parsedResponse = JSON.parse(match[1].trim());
} else {
  parsedResponse = JSON.parse(response); // fallback
}

⚡ Example: Creating a New Project

Here's what happens when you hit that "New Project" button:

1. 📝 You enter your Project Name and describe your vision

2. 🤖 DeepBuild sends your description to the AI with our special BRIEF_SYSTEM_PROMPT:

    const briefResponse = await sendMessage(
      `Create a project brief for the following utility app: ${description}`,
      'brief'
    );
   

3. 🎁 The AI returns structured JSON with everything we need

4. 💾 We store it all in IndexedDB with a fresh projectId

5. 💬 You can then chat with the AI to generate each file

Here's the code that kicks it all off:

const response = await sendMessage(
  `Create a project brief for a ${description}. Include implementation details...`,
  'brief'
);
const match = response.match(/<final_json>([^]*?)<\/final_json>/);
// ...
const parsedResponse = JSON.parse(match[1].trim());
// Then we create the project in IndexedDB via fileManager.saveProject

🌟 Contributing to DeepBuild

Ready to help make DeepBuild even more awesome? Here are some cool ways to contribute:

1. 🎯 New AI prompts: Got ideas for framework-specific prompts?

2. 🛠️ Error handling: Help us make the system more robust

3. 💅 UI/UX improvements: Add new language support, better diffs, or other dev-friendly features

4. 🔌 Plugins: Help us build a system for specialized generators (Docker, AWS, etc.)

🤝 How to Contribute

1. 📦 Clone the repo and get those dependencies installed

2. 🌱 Create your feature branch

3. ✨ Make your magic happen

4. 🚀 Open a pull request and let our CI pipeline do its thing!

We're on a mission to make software generation not just possible, but absolutely delightful! Join us in building the future of AI-assisted development.

🎬 Wrapping Up

And there you have it, friends! That's the inside scoop on how DeepBuild works its magic. Remember:

1. 💭 Dream up your project

2. 🤖 Let AI craft the plan

3. 🔨 Guide the process until your codebase is ready!

Got questions? Want to collaborate? Drop me a line! And don't forget to smash that ⭐ button if you're thinking about contributing!

Thanks for joining me on this code adventure! Now go forth and build something awesome!

🚀 Martin Bowling

🔗 Quick Links

• 🚀 Try DeepBuild:

  • Template: DeepBuild on Replit

  • Hosted Version: deepbuild.app

• 💻 Source Code: GitHub Repository

• 🤖 Powered By:

  • DeepSeek AI - State-of-the-art code generation

  • Hyperbolic Labs - Decentralized inference

  • Replit - Development platform and hosting

The llms.txt specification, created by Jeremy Howard, introduces a standardized way for websites to provide AI-readable content through a /llms.txt file in their root directory. This file contains markdown-formatted summaries, documentation, and links to key resources, making it easier for AI models to understand and process website content during inference time. Today, I want to share the journey of building this tool and the technical decisions that went into it.

Understanding llms.txt

Before diving into the implementation, let's talk about what llms.txt is and why it matters. Similar to robots.txt, llms.txt is a proposed standard that helps AI models better understand website content. It provides AI-generated summaries of your website's pages in a structured format. Here's what a basic llms.txt file looks like:

# Example Website

> This is the homepage of Example.com, featuring our main products and services.

## Main Pages

- [/about](/about): About page describing our company history and mission
- [/products](/products): Comprehensive catalog of our product offerings
- [/contact](/contact): Contact information and support resources

Technical Architecture

I built two versions of this tool: a modern Next.js web application and a Python implementation using Gradio. The Next.js version offers a polished UI with real-time updates, while the Gradio version provides a simpler interface that's perfect for quick deployments and testing.

Let's look at both implementations, starting with the Next.js version. Here's how the sitemap discovery process works:

const handleSubmit = async (e: React.FormEvent) => {
  // ... initialization code ...

  try {
    const normalizedUrl = normalizeUrl(url);
    const robotsUrl = new URL('/robots.txt', normalizedUrl).toString();

    let sitemapUrls: string[] = [];

    try {
      const robotsTxt = await fetchWithProxy(robotsUrl);
      sitemapUrls = extractSitemapUrlsFromRobotsTxt(robotsTxt);
    } catch (error) {
      addDebugMessage(`Failed to fetch robots.txt: ${error.message}`);
    }

    // If no sitemaps found in robots.txt, try common locations
    if (sitemapUrls.length === 0) {
      const commonLocations = getCommonSitemapUrls(normalizedUrl);
      // ... sitemap discovery logic ...
    }
  }
  // ... rest of the implementation
}

Working with AI Models

One of the most exciting aspects of this project was integrating multiple AI providers. I chose to support both Hyperbolic and Groq, as they offer distinct and complementary advantages:

Hyperbolic specializes in decentralized AI computing by aggregating idle computing resources, making AI development more accessible and cost-effective. Their open-access platform enables individuals and organizations to train and host AI models collaboratively.

Groq focuses on delivering exceptional AI inference speed and efficiency through their Language Processing Unit (LPU) technology. Designed specifically for AI inference, Groq's hardware and software platform offers instant speed and energy efficiency at scale.

Here's how we generate summaries using each provider.

First, the Hyperbolic implementation:

export async function generateHyperbolicSummary(url: string, apiKey: string): Promise<string> {
  const response = await fetch('https://api.hyperbolic.xyz/v1/chat/completions', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${apiKey}`,
    },
    body: JSON.stringify({
      model: 'meta-llama/Meta-Llama-3.1-8B-Instruct',
      messages: [
        {
          role: 'user',
          content: `Generate a concise 1-sentence summary of the purpose of this webpage: ${url}`
        }
      ],
      max_tokens: 200,
      temperature: 0.7,
      top_p: 0.9,
      stream: false
    }),
  });

  const json = await response.json();
  return json.choices[0].message.content;
}

And here's the Groq implementation:

export async function generateGroqSummary(url: string, content: str, api_key: string): Promise<string> {
  const groq = new Groq({ apiKey });

  const completion = await groq.chat.completions.create({
    messages: [{
      role: 'user',
      content: `Generate a concise 1-sentence summary of this webpage content:\n\nURL: ${url}\n\nContent: ${content}`
    }],
    model: "llama-3.2-1b-preview",
    temperature: 0.7,
    max_tokens: 200
  });

  return completion.choices[0]?.message?.content || '';
}

Content Extraction Challenge

One of the biggest challenges was extracting clean, meaningful content from web pages. Raw HTML is messy and full of navigation elements, ads, and other noise. That's where the Markdowner API comes in - it helps us get clean, formatted content that's perfect for AI processing.

async function getPageContent(url: string, markdownerKey?: string): Promise<string> {
  const headers = {"Accept": "text/plain"};
  if (markdownerKey) {
    headers["Authorization"] = `Bearer ${markdownerKey}`;
  }

  const response = await fetch("https://md.dhr.wtf/", {
    method: "POST",
    headers,
    body: JSON.stringify({ url })
  });

  return response.text();
}

Python Implementation with Gradio

In addition to the Next.js version, I also built a Python implementation using Gradio for a more streamlined deployment option. The Gradio version offers a simple yet powerful interface for generating llms.txt files. Here's how the core processing pipeline works:

def process_website(
    url: str,
    hyperbolic_key: str = "",
    groq_key: str = "",
    markdowner_key: str = "",
    use_hyperbolic: bool = True,
    progress=gr.Progress()
) -> Tuple[str, str, List[str], str]:
    try:
        if not (use_hyperbolic and hyperbolic_key) and not (not use_hyperbolic and groq_key):
            return "Error: Please provide an API key for the selected AI provider", None, [], ""

        base_url = normalize_url(url)
        progress(0, desc="Initializing...")

        # Try robots.txt first
        sitemap_urls = []
        try:
            robots_url = urljoin(base_url, '/robots.txt')
            robots_content = fetch_with_proxy(robots_url)
            sitemap_urls = extract_sitemap_urls_from_robots(robots_content)
        except:
            pass

        # Process sitemaps and generate summaries
        progress(0.4, desc="Processing sitemaps...")

        # ... rest of processing logic ...

        return llms_txt, json.dumps(summaries), all_urls, llms_full_txt

    except Exception as e:
        print(f"Error in process_website: {str(e)}")
        return f"Processing failed: {str(e)}", None, [], ""

The Gradio interface is built with a clean, user-friendly design that includes features like:

with gr.Blocks(title="llms.txt Generator", theme=gr.themes.Soft()) as demo:
    gr.Markdown("""
    # llms.txt Generator 🤖✨
    Generate AI-powered llms.txt files for any website 🌐
    """)

    with gr.Row():
        url_input = gr.Textbox(
            label="Website URL",
            placeholder="Enter website URL"
        )
        markdowner_key = gr.Textbox(
            label="Markdowner API Key (Optional)",
            placeholder="For higher rate limits",
            type="password",
            container=True,
            scale=2
        )

    # AI Provider Selection
    with gr.Row():
        with gr.Column():
            use_hyperbolic = gr.Checkbox(
                label="Use Hyperbolic",
                value=True,
                interactive=True
            )
            hyperbolic_key = gr.Textbox(
                label="Hyperbolic API Key",
                type="password",
                visible=True,
                placeholder="Enter your Hyperbolic API key"
            )

        with gr.Column():
            use_groq = gr.Checkbox(
                label="Use Groq",
                value=False,
                interactive=True
            )
            groq_key = gr.Textbox(
                label="Groq API Key",
                type="password",
                visible=False,
                placeholder="Enter your Groq API key"
            )

One of the key features of the Gradio version is its robust sitemap processing and URL extraction:

def extract_urls_from_sitemap(content: str) -> List[str]:
    urls = []
    try:
        root = ET.fromstring(content)
        ns = {
            'ns': root.tag.split('}')[0].strip('{')
        } if '}' in root.tag else {}

        # Handle sitemap index
        if 'sitemapindex' in root.tag:
            for sitemap in root.findall('.//ns:loc', ns):
                try:
                    sitemap_content = fetch_with_proxy(sitemap.text.strip())
                    urls.extend(extract_urls_from_sitemap(sitemap_content))
                except Exception:
                    continue
        # Handle urlset
        else:
            for url in root.findall('.//ns:loc', ns):
                urls.append(url.text.strip())
    except ET.ParseError:
        pass
    return urls

The Gradio implementation offers several advantages over the web version:

1. Easy deployment to platforms like Hugging Face Spaces

2. Built-in rate limiting and progress tracking

3. Simplified dependency management

4. Automatic file download handling

5. Real-time processing feedback

Both versions of the tool complement each other - the Next.js version provides a polished web experience, while the Gradio version offers a more straightforward path for quick deployments and testing.

Implementation Details

The project includes several key features that make it production-ready:

• Automatic sitemap discovery through robots.txt and common locations

• Rate limiting to respect API constraints

• Progress tracking for long-running operations

• Error handling and retry logic

• Clean, modern UI with dark mode support

Future Improvements

While the current version is fully functional, I have several improvements planned:

• Support for more AI providers

• Batch processing for large websites

• Custom summary templates

• API endpoint for programmatic access

Conclusion

Building the llms.txt Generator has been an exciting journey into the intersection of web technologies and AI. It's open source and available on GitHub, and I'm looking forward to seeing how the community uses and improves it.

The llms.txt standard is still evolving, and tools like this will help shape how websites and AI models interact in the future. ## Getting Started

There are several ways to start using the llms.txt Generator:

1. Hosted Version: If you want to try it out immediately without any setup, visit llms.website for a fully managed version that's always up to date. This is the easiest way to get started and requires no technical knowledge.

2. Self-Hosting: For developers who want full control, you can self-host either the Next.js or Gradio version. The complete source code is available on GitHub.

3. Quick Deploy Options:

   • Next.js Version: Use our Next.js Replit template to get a web version running in minutes

   • Gradio Version: Try our Gradio Replit template for a Python-based implementation

   • Both templates can also be easily deployed to platforms like Hugging Face Spaces or your preferred hosting provider

Each option has its advantages:

• The hosted version at llms.website is perfect for content teams and non-technical users

• Self-hosting gives you complete control over the environment and API keys

• Platform deployments offer a middle ground with some customization while maintaining ease of use

Resources & Links

• Main Project

  • 🌐 Hosted Version: llms.website

  • 📦 GitHub Repository: github.com/martinbowling/llms.website

• Templates

  • ⚡ Next.js Template: Replit Template

  • 🐍 Gradio Template: Replit Template

Choose the option that best fits your needs and technical comfort level. The hosted version is recommended for most users, while the templates and self-hosting options are perfect for developers who want to customize the tool or integrate it into their own workflows.

If you have any questions about deployment options or need help getting started, feel free to reach out to me on Twitter or through the GitHub repository. I'm always happy to help users get started with whichever deployment option they choose!

The Evolution of AI Reasoning 📚

Before we sink our teeth into Llamaberry, let's take a quick trip down memory lane. AI reasoning has come a long way, baby!

• 1950s-1960s: Rule-based systems. Think of these as the "If This, Then That" of the AI world.

• 1980s-1990s: Expert systems. These were like having a virtual expert in a specific field.

• 2000s-2010s: Machine learning models. Now we're talking! AI that can learn from data.

• 2020s: Large Language Models (LLMs) and Chain of Thought reasoning. This is where things get really exciting!

Chain of Thought reasoning is like giving AI a notepad to show its work. Instead of just spitting out an answer, it walks us through its thinking process. And that's where Llamaberry comes in! 🌟

What's Llamaberry, You Ask? 🤔

Imagine if you could teach an AI to think not just fast, but thoroughly - like a human expert pondering a complex problem. That's exactly what I set out to do with Llamaberry. It's my implementation of a multi-turn Chain of Thought (CoT) reasoning system, powered by the Llama 3.1 70B model on Groq. Fancy name aside, it's all about making AI think more like us... only faster and without needing coffee breaks! ☕

The Secret Sauce: Multi-Turn Reasoning 🥣

Here's where it gets interesting. Instead of asking the AI to give us an answer in one go, we're asking it to think through the problem in multiple steps, or "turns." It's like watching a master chef prepare a complex dish - each step builds on the last, resulting in something far more refined than a quick microwave meal.

Let's break it down:

1. Turn 1 🥚: The AI takes a first crack at the problem.

2. Turn 2 🍳: It looks back at its first attempt and tries to improve or refine its thinking.

3. Turn 3 🧑‍🍳: One more round of reflection and refinement.

4. Synthesis 🍽️: Finally, it puts all these thoughts together into one coherent, well-reasoned answer.

To visualize this process, here's a delightful little diagram that shows how each turn builds on the previous one:

    Turn 1         Turn 2         Turn 3
   🧠💡📝         🔍🤔📊         🔬🧪🎯
 ┌─────────┐    ┌─────────┐    ┌─────────┐
 │  Input  │    │ Output  │───▶│  Input  │
 └─────────┘    └─────────┘    └─────────┘
      │              ▲              |
      │              │              │
      ▼              │              ▼
 ┌─────────┐    ┌─────────┐    ┌─────────┐
 │Reasoning│    │Reasoning│    │Reasoning│
 └─────────┘    └─────────┘    └─────────┘
      │              ▲              │
      │              │              │
      ▼              │              ▼
 ┌─────────┐    ┌─────────┐    ┌─────────┐
 │ Output  │───▶│  Input  │    │ Output  │
 └─────────┘    └─────────┘    └─────────┘
                                    │
                                    │
                                    ▼
                               ┌─────────┐
                               │Synthesis│
                               └─────────┘
                                    │
                                    │
                                    ▼
                            🎉 Final Answer 🎉

Isn't that neat? You can see how each turn's output becomes the input for the next turn, allowing the AI to refine its thinking at each stage. And at the end, all of this thoughtful pondering gets distilled into one final, well-reasoned answer. It's like watching a berry ripen to perfection! 🍓

The Recipe: How It All Works 📝

Now, let's get into the nitty-gritty. Don't worry; I promise to keep it as smooth as a berry smoothie!

Step 1: Setting the Stage 🎭

We start by setting up our AI assistant with some ground rules. Here's a snippet of what that looks like:

initial_system_prompt = """You are an AI assistant capable of detailed, step-by-step thinking. When presented with a question or problem, break down your thought process into clear, logical steps. For each step, explain your reasoning. Conclude with a final answer. Use the following markdown structure:

## Reasoning
1. [First step]
   **Explanation:** [Detailed explanation of this step]
2. [Second step]
   **Explanation:** [Detailed explanation of this step]
...

## Answer
[Final answer]

Be comprehensive and show your reasoning clearly."""

This is like giving our AI chef a recipe to follow. It knows it needs to break down its thinking into steps and explain each one, all neatly formatted in markdown.

Step 2: The Thinking Process 🤔💭

For each turn of reasoning, we ask the AI to think about the problem. But here's the cool part - after the first turn, we also ask it to consider what it thought before. It's like asking a friend, "Hey, remember what you said earlier? Let's think about that again."

Here's how we generate each turn:

async def generate_turn(query: str, previous_turns: list = None) -> str:
    is_first_turn = previous_turns is None or len(previous_turns) == 0
    if is_first_turn:
        messages = [{
            "role": "system",
            "content": initial_system_prompt
        }, {
            "role": "user",
            "content": query
        }]
    else:
        previous_content = "\n\n".join(previous_turns)
        messages = [{
            "role": "system",
            "content": followup_system_prompt
        }, {
            "role":
            "user",
            "content":
            f"Original Query: {query}\n\nPrevious Turns:\n{previous_content}\n\nProvide the next turn of reasoning."
        }]

    return await call_llm(messages)

This function is the heart of our multi-turn reasoning. It decides whether we're on the first turn or a subsequent one, and structures the prompt accordingly. For turns after the first, it includes all previous turns in the context, allowing the AI to build on its earlier thoughts.

Step 3: Putting It All Together 🧩

After three turns of thinking, we ask our AI to look at all its thoughts and come up with a final, well-reasoned answer. It's like asking a group of experts to discuss a problem and then write a report together.

async def synthesize_turns(query: str, turns: list) -> str:
    turns_text = "\n\n".join(
        [f"Turn {i+1}:\n{turn}" for i, turn in enumerate(turns)])
    messages = [{
        "role": "system",
        "content": synthesis_prompt
    }, {
        "role":
        "user",
        "content":
        f"Original Query: {query}\n\nTurns of Reasoning:\n{turns_text}"
    }]
    return await call_llm(messages)

This synthesis step is crucial. It's where our AI takes all its thoughts from the previous turns and distills them into a coherent, final answer.

Single-Turn vs Multi-Turn: A Tasty Comparison 🍎🍏

Let's compare single-turn and multi-turn reasoning with a simple example. Imagine we ask the AI: "What are the potential impacts of artificial intelligence on the job market?"

Single-Turn Reasoning:

AI: Artificial intelligence will likely have significant impacts on the job market, including:
1. Automation of routine tasks
2. Creation of new job roles in AI development and maintenance
3. Potential job displacement in certain industries
4. Increased demand for skills in data analysis and machine learning

While AI may eliminate some jobs, it will also create new opportunities and increase productivity in many fields.

Multi-Turn Reasoning with Llamaberry:

Turn 1:

AI: Let's approach this step-by-step:

1. Identify current AI capabilities
   Explanation: AI can perform data analysis, pattern recognition, and automate routine tasks.

2. Analyze vulnerable job sectors
   Explanation: Manufacturing, customer service, and data entry are likely to be affected.

3. Consider new job opportunities
   Explanation: AI development, data science, and AI ethics consultation may see growth.

4. Examine potential economic effects
   Explanation: Increased productivity might lead to economic growth, but wealth distribution could be a challenge.

Initial Answer: AI will likely cause significant shifts in the job market, with both job losses in some sectors and new opportunities in others.

Turn 2:

AI: Let's refine our thinking:

1. Reassess the timeline of AI impact
   Explanation: The effects won't be immediate but will unfold over years or decades.

2. Consider the adaptability of the workforce
   Explanation: Retraining and education will be crucial for workers to transition to new roles.

3. Analyze potential societal changes
   Explanation: The nature of work itself might change, with more emphasis on creativity and emotional intelligence.

4. Examine global differences
   Explanation: AI's impact will vary across different countries and economies.

Refined Answer: AI's impact on the job market will be complex and gradual, requiring proactive measures in education and economic policy to ensure a smooth transition.

Turn 3:

AI: Let's delve deeper:

1. Explore the concept of human-AI collaboration
   Explanation: Many jobs may evolve to involve working alongside AI rather than being replaced by it.

2. Consider the ethical implications
   Explanation: Issues like algorithmic bias in hiring processes need to be addressed.

3. Analyze the potential for new economic models
   Explanation: Universal Basic Income or reduced working hours might become more prominent discussions.

4. Examine historical parallels
   Explanation: Previous technological revolutions, like the Industrial Revolution, can offer insights into potential outcomes.

Further Refined Answer: While AI will undoubtedly transform the job market, its impact will likely be nuanced, involving collaboration between humans and AI, ethical considerations, and potentially new economic paradigms.

Synthesis:

AI: After careful consideration across multiple turns of reasoning, we can synthesize our thoughts on the potential impacts of artificial intelligence on the job market:

1. Job Transformation: AI will lead to significant changes in the job market, but these changes will be gradual and complex. While some jobs may be automated, many others will be transformed to involve collaboration between humans and AI. Industries like manufacturing, customer service, and data entry are likely to see the most immediate impacts.

2. New Opportunities: The rise of AI will create new job categories, particularly in fields like AI development, data science, AI ethics consultation, and human-AI interaction design. There will likely be an increased demand for skills in creativity, emotional intelligence, and complex problem-solving that AI cannot easily replicate.

3. Economic and Social Implications: The integration of AI into the workforce has the potential to increase productivity and economic growth. However, it also raises concerns about wealth distribution and may necessitate discussions about new economic models, such as Universal Basic Income or reduced working hours.

4. Education and Adaptation: The changing job market will require a strong emphasis on continuous learning and adaptation. Educational systems and workforce training programs will need to evolve to prepare people for the new reality of working alongside AI.

5. Ethical Considerations: As AI becomes more prevalent in hiring and work processes, addressing issues like algorithmic bias and ensuring fair opportunities for all workers will be crucial.

6. Global Variations: The impact of AI on the job market will not be uniform across the globe. Different countries and economies will experience varying effects based on their economic structures, technological readiness, and policy responses.

7. Historical Context: While AI represents a significant shift, it's important to view it in the context of previous technological revolutions. Like the Industrial Revolution, the AI revolution will likely lead to short-term disruptions but potentially long-term benefits if managed properly.

In conclusion, while AI will undoubtedly have profound impacts on the job market, these impacts will be multifaceted and evolve over time. The key to navigating this transition successfully lies in proactive measures in education, economic policy, and ethical governance, ensuring that the benefits of AI are broadly shared while mitigating potential negative effects.

As you can see, the multi-turn approach allows for a much more nuanced, comprehensive analysis of the question. It considers multiple angles, refines its thinking, and produces a more thoughtful and well-rounded final answer.

Why Is This Cool? 😎

1. Depth of Thought: By thinking in multiple turns, the AI can dig deeper into problems, considering angles it might miss in a single pass.

2. Self-Improvement: The AI gets to critique and improve its own thinking, just like we do when we mull over a problem.

3. Transparency: We can see the AI's thought process, making it easier to understand and trust its conclusions.

4. Flexibility: This approach can handle a wide range of questions, from simple queries to complex problems that require nuanced thinking.

5. Markdown Magic: By using markdown formatting, our AI's thoughts are not just thorough, but also beautifully structured and easy to read!

Real-World Applications 🌍

The multi-turn reasoning approach of Llamaberry has numerous potential applications across various fields:

1. Medical Diagnosis: Doctors could use this system to get a second opinion, with the AI considering multiple angles of a patient's symptoms and medical history.

2. Legal Analysis: Lawyers could use it to analyze complex cases, considering multiple interpretations of the law and previous case precedents.

3. Scientific Research: Researchers could use it to generate hypotheses, considering multiple angles and potential experimental approaches.

4. Business Strategy: Executives could use it to analyze market trends and make strategic decisions, considering multiple scenarios and potential outcomes.

5. Education: Teachers could use it to create personalized learning plans, considering multiple aspects of a student's learning style and needs.

Limitations and Ethical Considerations 🚧

While Llamaberry and similar multi-turn reasoning systems are powerful, it's important to acknowledge their limitations and potential ethical concerns:

1. Bias: Like all AI systems, Llamaberry can potentially perpetuate biases present in its training data.

2. Hallucination: Even with multiple turns, AI can sometimes generate plausible-sounding but incorrect information.

3. Computational Cost: Multi-turn reasoning requires more computational resources than single-turn approaches.

4. Over-reliance: There's a risk of people relying too heavily on AI for decision-making without appropriate human oversight.

5. Privacy: When used with sensitive information, proper data protection measures must be in place.

How to Get Started 🚀

Want to try out Llamaberry for yourself? I've got great news – it's now easier than ever! Thanks to the magic of Replit, you can start experimenting with Llamaberry in just a few clicks. Here's how:

1. Head to Replit: Click on this link to access the Llamaberry template: https://replit.com/@MartinBowling/Llamaberry-Powered-By-Groq?v=1

2. Fork the Template: Once you're on the Replit page, click the "Fork" button to create your own copy of the Llamaberry project.

3. Get Your Groq API Key: If you haven't already, sign up for a Groq account and obtain your API key.

4. Set Up Your Environment: In your forked Replit project, find the "Secrets" tab (it looks like a lock icon). Add a new secret with the key GROQ_API_KEY and your Groq API key as the value.

5. Run the Project: Click the "Run" button at the top of the Replit interface. This will start up the Llamaberry application.

6. Start Experimenting: Once the application is running, you'll see a Gradio interface where you can input your questions and see Llamaberry's multi-turn reasoning in action!

That's it! No need to worry about Python versions, installing dependencies, or configuring your local environment. Replit takes care of all that for you, so you can focus on the fun part – exploring AI reasoning!

Remember, the output will be in beautiful markdown format, making it easy to read and understand the AI's thought process. You can ask questions, analyze the reasoning steps, and even modify the code to experiment with different prompts or parameters.

So what are you waiting for? Dive in and start your AI reasoning adventure with Llamaberry! 🚀🧠🍓

Wrapping Up 🎁

And there you have it, folks! Llamaberry is my attempt at making AI reasoning a bit more... well, reasonable. It's not perfect (what is?), but it's a step towards AI that can think more like us humans - iteratively, reflectively, and thoroughly. And now, with markdown formatting, it's not just smart thinking, it's pretty thinking too!

Next time you're pondering a tough question, remember: even AIs sometimes need to sleep on it (or at least think about it three times) before coming up with their best answer. And isn't that just deliciously human of them?

Until next time, keep your thoughts flowing and your berries sweet!

Today we're diving into an intriguing development in the world of artificial intelligence: Speculative Retrieval-Augmented Generation, commonly known as Speculative RAG. This technique represents an interesting step forward in how AI systems can process and respond to complex queries. Let's explore how Speculative RAG works and why it's garnering attention in AI circles.

What's the Big Deal with Speculative RAG?

Imagine you're at a party, and someone asks a question. You've got a few friends there: the Quick Thinker, the Bookworm, and the Judge. Speculative RAG is like having these three friends work together to give the best possible answer. Let's break it down:

1. The Quick Thinker (Generalist LLM): This is the friend who quickly decides if the question needs a simple answer or if it's time to call in the big guns.

2. The Bookworm (Specialist LLM): If the question is tricky, this friend dives into the books (or in our case, the provided documents) and comes up with detailed answers.

3. The Judge (Evaluator LLM): This friend listens to all the answers from the Bookworm and picks the best one.

Now, let's take a look at the technical implementation of Speculative RAG and see how these components work together in practice.

The Code: Where the Magic Happens

Let's walk through the key parts of our Speculative RAG implementation. Don't worry if you're not a coding guru – I'll break it down for you!

Setting the Stage

First, we need to import our tools and set up our AI models:

import os
import asyncio
from groq import AsyncGroq
from typing import List, Tuple

# Initialize Groq client
client = AsyncGroq(api_key=os.environ.get("GROQ_API_KEY"))

# Define models
generalist_model = "llama-3.1-8b-instant"
specialist_model = "mixtral-8x7b-32768"

Here, we're using the Groq API to access our AI models. The generalist is quick on its feet, while the specialist is the heavyweight champion of knowledge.

The Quick Thinker in Action

async def generalist_llm(query: str) -> Tuple[bool, str]:
    messages = [
        {"role": "system", "content": generalist_system_prompt},
        {"role": "user", "content": query}
    ]
    response = await call_llm(generalist_model, messages, temperature=0.5, max_tokens=10)
    is_complex = response.lower().strip() == 'yes'
    return is_complex, f"Generalist decision: {'Knowledge-intensive' if is_complex else 'Simple'}"

This function is our Quick Thinker. It looks at the question and decides if we need to call in the Bookworm or if it can handle it solo. The temperature=0.5 keeps it focused, and max_tokens=10 ensures a quick yes/no response.

The Bookworm's Deep Dive

async def specialist_llm(query: str, document: str) -> Tuple[str, str, str]:
    messages = [
        {"role": "system", "content": specialist_system_prompt},
        {"role": "user", "content": f"Query: {query}\n\nDocument:\n{document}"}
    ]
    response = await call_llm(specialist_model, messages, temperature=0.8, max_tokens=2048)

    parts = response.split("Rationale:", 1)
    draft = parts[0].strip()
    rationale = parts[1].strip() if len(parts) > 1 else "No explicit rationale provided."

    return draft, rationale, f"Specialist generated a draft."

Here's where the Bookworm shines. It takes the question and a document, then generates a detailed answer (draft) and explains its thinking (rationale). We use a higher temperature=0.8 to encourage creative thinking and allow for longer responses with max_tokens=2048.

The Judge's Wise Decision

async def evaluator_llm(query: str, drafts_and_rationales: List[Tuple[str, str]]) -> Tuple[int, str, str]:
    drafts_text = "\n\n".join([f"Draft {i+1}:\n{draft}\nRationale:\n{rationale}" 
                               for i, (draft, rationale) in enumerate(drafts_and_rationales)])
    messages = [
        {"role": "system", "content": evaluator_system_prompt},
        {"role": "user", "content": f"Query: {query}\n\nDrafts and Rationales:\n{drafts_text}"}
    ]
    response = await call_llm(generalist_model, messages, temperature=0.3, max_tokens=512)

    lines = response.split('\n')
    best_draft_num = int(lines[0].split(':')[1].strip()) - 1
    rationale = '\n'.join(lines[1:]).strip()

    return best_draft_num, rationale, f"Evaluator selected Draft {best_draft_num + 1} as the best."

The Judge (Evaluator) reviews all the drafts, picks the best one, and explains why. We use a lower temperature=0.3 here because we want a more deterministic, focused decision.

Putting It All Together

The heart of our Speculative RAG system is the speculative_rag function:

async def speculative_rag(query: str, documents: List[str]) -> Tuple[str, str]:
    process_log = []

    # Step 1: Quick Thinker decides
    is_complex, gen_log = await generalist_llm(query)
    process_log.append(gen_log)

    if is_complex:
        # Step 2: Bookworm generates drafts
        tasks = [process_document(query, doc) for doc in documents]
        drafts_and_rationales = await asyncio.gather(*tasks)
        process_log.append(f"Specialist generated {len(drafts_and_rationales)} drafts.")

        # Step 3: Judge selects the best draft
        best_draft_num, eval_rationale, eval_log = await evaluator_llm(query, drafts_and_rationales)
        process_log.append(eval_log)

        # Step 4: Craft final response using the best draft
        best_draft = drafts_and_rationales[best_draft_num][0]
        final_response, final_log = await final_response_llm(query, best_draft, eval_rationale)
        process_log.append(final_log)
    else:
        # For simple queries, Quick Thinker handles it
        final_response = await call_llm(generalist_model, [
            {"role": "system", "content": "You are a helpful assistant. Please answer the following query concisely."},
            {"role": "user", "content": query}
        ], temperature=0.7, max_tokens=512)
        process_log.append(f"Simple query: Generalist provided the response.")

    return final_response, "\n".join(process_log)

This function orchestrates the whole show. It's like the party host making sure everyone plays their part:

1. The Quick Thinker assesses the question.

2. If it's complex, the Bookworm generates answers for each document.

3. The Judge picks the best answer.

4. We use the best answer to craft a final, polished response.

If the question is simple, the Quick Thinker handles it directly. Efficiency at its finest!

Why This is Cool

Speculative RAG is like having a team of AI experts working together to answer your questions. It's smart enough to handle simple queries quickly and to dive deep when needed. Plus, it can use multiple documents to generate comprehensive answers.

The best part? It's all happening behind the scenes. To the user, it looks like they're just asking a question and getting a great answer. But we know the truth – there's a whole party of AI friends working hard to make it happen!

Wrapping Up

And there you have it, folks! We've just taken a whirlwind tour of Speculative RAG. We've seen how it combines the speed of a quick-thinking generalist with the depth of a knowledgeable specialist and the wisdom of a fair judge.

This approach opens up exciting possibilities for more efficient, accurate, and context-aware AI interactions. Whether you're building a chatbot, a question-answering system, or just exploring the frontiers of AI, Speculative RAG is definitely a technique worth keeping in your toolbox.

Remember, the key to great AI is not just about having a big brain – it's about knowing when to think quick, when to dig deep, and when to call for a second opinion. Speculative RAG does all of that and more!

So, next time you're chatting with an AI and it gives you a surprisingly good answer, you can smile and think, "I know what's going on behind the curtain!" And who knows? Maybe you'll be inspired to create your own AI party with Speculative RAG as the guest of honor.

Until next time, keep coding, keep learning, and keep pushing the boundaries of what's possible with AI!

Grab the code at replit

Happy Friday 🎉 I'm excited to share a project I've been working on that combines two powerful tools: Highlight and MultiOn. 🛠️💡But before we dive into the code, I want to take a moment to highlight some amazing opportunities for developers in the Highlight ecosystem.

Getting Started with Highlight

First off, if you're not already familiar with Highlight, their comprehensive developer docs are an absolute goldmine of information. Whether you're just getting started or looking to level up your Highlight skills, these docs are your go-to resource.

For those who love to connect and collaborate, the Highlight Discord community is the place to be. It's a vibrant hub where developers share ideas, troubleshoot issues, and showcase their latest creations.

Highlight Grant Program

Now, here's something that might really pique your interest: Highlight's incredible grant program for developers. They're offering $10,000 per developer, with $5,000 upfront and another $5,000 upon app publication. The best part? No equity is taken! But wait, there's more:

• Free office space in NYC's Upper West Side whenever you need it (perfect for those intense coding sessions or team meetups)

• An additional $20,000 follow-on grant available, depending on your app's post-launch traction

If you're ready to take your development journey to the next level with Highlight, you can sign up for the grant program here.

Introducing MultiOn: Your AI Agent API

To get started with this project, you'll also need a MultiOn API key. MultiOn offers an Agent API that allows you to build and embed AI agents that can complete tasks and workflows on the web for your users. You can get your API key here.

What MultiOn Offers:

• Versatile Actions: From ordering on Amazon to sending calendar invites, booking flights, or controlling the AWS console, MultiOn's general-purpose agents can handle a wide range of tasks.

• Cross-Sector Applications: Whether you're in e-commerce, travel, financial services, or developing smart devices, MultiOn can automate tasks and assist customers.

• Seamless Integration: MultiOn allows users to complete purchases or bookings without leaving your website, fill complex forms, and aggregate data from various online sources.

With these resources and support at your fingertips, the possibilities are endless. And speaking of possibilities, let me show you what I've been cooking up with Highlight and MultiOn...

The Power of Context-Aware Computing

Before we dive into the code, let's talk about why this project is so cool. We're living in an age where our computers can understand what's on our screens and act on that information in real-time. That's where Highlight comes in – it's a game-changing platform that allows developers to create apps that can "see" and interpret what's on a user's screen.

Pair that with MultiOn, a tool that can interact with web pages programmatically, and you've got a recipe for some seriously powerful applications.

Introducing Secret Shopper

Secret Shopper is a background shopping tool that works quietly while you browse. It detects when you're looking at a product, then automatically fetches price comparisons from other retailers. Let's break down how it works:

1. Screen Content Analysis

The heart of Secret Shopper is its ability to understand what's on your screen. We use Highlight's OCR capabilities to read the screen content, then pass that to a language model for classification. Here's how it works in detail:

export async function classifyProductPage(ocrContent: string): Promise<ProductClassificationResult> {
  try {
    const messages: LLMMessage[] = [
      {
        role: "system",
        content: "You are an AI trained to analyze OCR content and determine if it's from a product page..."
      },
      {
        role: "user",
        content: `Analyze this OCR content: ${ocrContent}`
      }
    ];

    const textPrediction = await Highlight.inference.getTextPrediction(messages);
    let fullResponse = '';
    for await (const chunk of textPrediction) {
      fullResponse += chunk;
    }

    const jsonMatch = fullResponse.match(/\{[\s\S]*\}/);

    if (jsonMatch) {
      const jsonString = jsonMatch[0];
      return JSON.parse(jsonString);
    } else {
      throw new Error('Failed to extract JSON from the response');
    }
  } catch (error) {
    console.error('Error classifying the page:', error);
    throw new Error('Failed to classify the page.');
    }
}

This function does several important things:

1. It sets up a conversation with the AI model, giving it instructions on how to analyze the OCR content.

2. It uses Highlight's getTextPrediction method to get a response from the AI.

3. It processes the response, extracting the JSON part that contains the classification results.

We also have an alternative implementation using the Groq API, which shows how you can easily swap out different AI providers:

export async function classifyProductPageGroq(ocrContent: string): Promise<ProductClassificationResult> {
  try {
    const chatCompletion = await groq.chat.completions.create({
      messages: [
        {
          role: "system",
          content: "You are an AI trained to analyze OCR content..."
        },
        {
          role: "user",
          content: `Analyze this OCR content: ${ocrContent}`
        }
      ],
      model: "llama-3.1-70b-versatile",
      temperature: 0.5,
      max_tokens: 1024,
      top_p: 1,
      stream: false,
      stop: null
    });

    // ... process and return the result
  } catch (error) {
    console.error('Error classifying the page:', error);
    throw new Error('Failed to classify the page.');
  }
}

2. Fetching Price Comparisons

Once we know we're looking at a product, it's time to fetch some price comparisons. This is where MultiOn comes in:

export async function retrieveProductListings(product: string, multionKey: string): Promise<ProductListing[]> {
  const multion = new MultiOnClient({ apiKey: multionKey });

  try {
    const response = await multion.retrieve({
      cmd: `Get all non-sponsored listings on Google for ${product} with the price, description, and URL...`,
      url: `https://www.google.com/search?q=${encodeURIComponent(product)}`,
      fields: ["price", "description", "url"]
    });

    // ... process and return the results
  } catch (error) {
    console.error("Error retrieving product listings:", error);
    throw new Error("Failed to retrieve product listings.");
  }
}

This function uses MultiOn to search for our product and return a list of prices and retailers. It's like having a little web-scraping genie at your command! 🧞‍♂️

3. Managing State with React Context

We use React's Context API to manage the state of our application. Here's a snippet from our ProductContext:

export const ProductProvider: React.FC<{ children: React.ReactNode }> = ({ children }) => {
  const [productGroups, setProductGroups] = useState<ProductGroup[]>([]);
  const [checkInterval, setCheckInterval] = useState(60);
  const [multionKey, setMultionKey] = useState('');
  // ... other state variables

  const addProductGroup = useCallback((name: string, products: ProductListing[]) => {
    setProductGroups((prevGroups) => {
      const existingGroup = prevGroups.find((group) => group.name === name);
      if (existingGroup) {
        // Update existing group
      } else {
        // Add new group
      }
    });
  }, []);

  // ... other methods

  const fetchProductUpdates = useCallback(async () => {
    try {
      const context = await Highlight.user.getContext();
      const ocrContent = context.environment?.ocrScreenContents || '';

      if (ocrContent) {
        const classification = await classifyProductPage(ocrContent);

        if (classification.isProductPage || classification.isProductDiscussion || classification.isProductMention) {
          const productName = classification.productName;

          if (productName) {
            // Fetch and store product listings
            const listings = await retrieveProductListings(productName, multionKey);
            addProductGroup(productName, listings);
          }
        }
      }
    } catch (error) {
      console.error('Error fetching product updates:', error);
    }
  }, [productGroups, addProductGroup, multionKey, backgroundPermissionGranted]);

  // ... context value and provider
};

This context manages the state of our application, including the list of product groups, settings, and the main fetchProductUpdates function that ties everything together.

4. The User Interface

Our main application component, SecretShopperApp, uses this context to render the UI and manage the background processing:

const SecretShopperApp: React.FC = () => {
  const { 
    productGroups, 
    removeProductGroup, 
    fetchProductUpdates, 
    checkInterval,
    showOnboarding,
    setShowOnboarding
  } = useContext(ProductContext);

  const [isPaused, setIsPaused] = useState(false);

  useEffect(() => {
    let intervalId: NodeJS.Timeout | null = null;

    if (!isPaused) {
      intervalId = setInterval(() => {
        fetchProductUpdates();
      }, checkInterval * 1000);
    }

    return () => {
      if (intervalId) clearInterval(intervalId);
    };
  }, [fetchProductUpdates, checkInterval, isPaused]);

  // ... render UI
};

This component sets up an interval to fetch product updates regularly, but only when the app isn't paused. It's a simple way to give users control while still leveraging the power of background processing.

The Power of Background Processing

One of the coolest things about Secret Shopper is that it runs in the background, thanks to Highlight's background permission feature. This is managed in our OnboardingScreen component:

const requestBackgroundPermission = useCallback(async () => {
  try {
    const granted = await Highlight.permissions.requestBackgroundPermission();
    if (granted) {
      await Highlight.app.setBackgroundStatus(true);
      setBackgroundPermissionGranted(true);
      await saveToAppStorage("backgroundPermissionGranted", "true");
      setError(null);
    } else {
      setError("Background permission was not granted. This app requires background access to function properly.");
    }
    return granted;
  } catch (error) {
    console.error('Error requesting background permission:', error);
    setError("An error occurred while requesting background permission. Please try again.");
    return false;
  }
}, []);

This function requests background permission from the user and sets up the app to run in the background if granted.

Wrapping Up

Secret Shopper demonstrates the power of context-aware computing. By leveraging Highlight's screen analysis capabilities and MultiOn's web interaction abilities, we've created an app that understands what we're looking at and acts on that information automatically.

The possibilities for this kind of technology are endless. Imagine an app that automatically takes notes in your meetings, or one that suggests relevant documents based on what you're working on. The future of computing is context-aware, and tools like Highlight and MultiOn are making it possible for developers to build these kinds of intelligent, responsive applications.

That's all for now, folks! Feel free to dive into the code, experiment with it, and see what you can build. Happy coding! 👨‍💻🚀

Why Content Moderation Matters 🤔

In the world of AI and large language models, it's crucial to ensure that our digital conversations remain respectful and safe. That's where content moderation comes in. It helps us filter out potentially harmful or inappropriate content, making sure our AI assistants behave responsibly. Think of it as a friendly bouncer for your chat app! 🕴️

Enter Llama Guard 3 🦙

Llama Guard 3 is our superhero in this story. It's a powerful 8B parameter model designed to classify content in both AI inputs and outputs. It's like having a wise old llama watching over our conversations, making sure everything stays friendly and appropriate. 🦸‍♂️

Let's Build Our Safe Chat App! 🛠️

I've put together a fun little project using Python and the Gradio library to create a web-based chat interface. Here's a sneak peek at how we set things up:

import os
import asyncio
import gradio as gr
from groq import AsyncGroq

client = AsyncGroq(api_key=os.environ.get("GROQ_API_KEY"))

chat_model = "llama3-70b-8192"
moderation_model = "llama-guard-3-8b"

system_prompt = """You are a helpful, respectful, and honest assistant. Always strive to answer the user's questions to the best of your ability, while maintaining a polite and professional tone."""

This sets up our connection to the Groq API and defines which models we'll use for chat and moderation. The system prompt gives our AI assistant its personality – think of it as its morning pep talk! ☀️

The Magic of Moderation ✨

Here's where Llama Guard 3 really shines. We use it to check each message before our main AI responds:

async def moderate_message(message):
    """Moderate the user message using Llama Guard 3."""
    response = await client.chat.completions.create(
        model=moderation_model,
        messages=[{"role": "user", "content": message}],
        temperature=0.1,
        max_tokens=10,
    )
    return response.choices[0].message.content.strip().lower() == "safe"

If Llama Guard 3 gives us the thumbs up, we proceed with generating a response. If not, we politely decline to engage with the unsafe content. It's like having a wise friend who knows when to change the subject! 😉

Streaming Responses for a Smooth Experience 🌊

To make our chat feel more natural and responsive, we stream the AI's responses:

async def chat_response(message, history):
    """Generate a chat response, including moderation."""
    is_safe = await moderate_message(message)

    if not is_safe:
        yield "I apologize, but I can't respond to that type of message. Let's keep our conversation respectful and appropriate."
        return

    # ... (code for generating and streaming the response)

This way, the response appears gradually, just like a real person typing. It's much more engaging than waiting for the whole message to pop up at once!

A Pretty Interface with Gradio 🎨

Gradio makes it super easy to create a good-looking web interface for our chat app. Here's a snippet of how we set it up:

with gr.Blocks(theme=gr.themes.Soft(), fill_height=True, head=js) as demo:
    gr.Markdown("# SafeChat: Moderated AI Conversation")
    gr.Markdown("This chat interface uses Llama Guard 3 to moderate messages and ensure safe interactions.")

    chatbot = gr.Chatbot(height=450, show_label=False)
    message = gr.Textbox(
        placeholder="Type your message here...",
        label="User Input",
        show_label=False,
        container=False
    )
    submit = gr.Button("Send", variant="primary")
    clear = gr.Button("Clear")

    # ... (code for handling user input and bot responses)

This creates a clean, user-friendly interface with a chat window, input box, and buttons for sending messages and clearing the chat.

Try It Out! 🚀

I've made this project a template on Replit, so you can give it a spin right now! Just head over to Replit and start chatting. Feel free to ask about quantum physics, plan a trip to Japan, or discuss the finer points of machine learning algorithms. Our AI assistant is ready for a fun and safe conversation!

Wrapping Up 🎁

Building this safe chat experience has been a blast, and I hope you enjoy using it as much as I enjoyed creating it. It's amazing to see how we can combine cutting-edge AI technology with responsible safeguards to create something both powerful and trustworthy.

Remember, in the world of AI, with great power comes great responsibility. By using tools like Llama Guard 3, we can ensure that our AI interactions remain positive, informative, and safe for everyone.

Happy chatting, and may your conversations be ever insightful and respectful! 🌟💬

Pop into their discord to join the ever growing community of Groq devs. The future is bright and thanks to Groq's LPU's it 🚀🚀🚀🚀🚀

The Challenge: Balancing Power and Efficiency

Large language models like Llama 3.1 70B are incredibly powerful, but they come with significant computational costs. On the other hand, smaller models like Llama 3.1 8B are more efficient but may lack some of the capabilities of their larger counterparts. The challenge lies in bridging this gap – how can we leverage the power of larger models to optimize prompts for smaller, more efficient ones?

The Solution: A Prompt Optimization Pipeline

My tool addresses this challenge through a sophisticated prompt optimization pipeline. Here's how it works:

1. Task Description: Users input a description of the task they want to accomplish.

2. Initial Prompt Generation: The tool uses the 70B model to generate an initial prompt based on the task description.

3. Prompt Variation: It then creates multiple variations of the prompt, aiming for diversity and improvement.

4. System Prompt Generation: A system prompt is created to guide the 8B model in understanding the task.

5. 8B Model Testing: The tool tests the optimized prompts on the 8B model, providing detailed results.

Let's dive into the key components of this tool:

Generating the Initial Prompt

def generate_prompt(task):

completion = client.chat.completions.create(

model=MODEL_70B,

messages=[{

"role": "system",

"content": "You are an AI assistant that generates example prompts for given tasks. Respond with only the prompt, nothing else."

}, {

"role": "user",

"content": f"Generate a sample prompt for the following task: {task}"

}],

temperature=0.7,

max_tokens=500,

)

return completion.choices[0].message.content.strip()

This function uses the 70B model to generate an initial prompt based on the user's task description. It's designed to be focused and concise, providing a solid starting point for optimization.

Creating Prompt Variations

def generate_candidate_prompts(task, prompt_example):

completion = client.chat.completions.create(

model=MODEL_70B,

messages=[{

"role": "system",

"content": "Given an example prompt, create seven additional prompts for the same task that are even better. Ensure the new prompts are diverse and unique from one another."

}, {

"role": "user",

"content": f"Task: {task}\nExample prompt: {prompt_example}"

}],

temperature=0.5,

max_tokens=4000,

)

prompts = completion.choices[0].message.content.strip().split('\n')

return prompts[:7] # Ensure we only return 7 prompts

This function generates multiple variations of the initial prompt, aiming for diversity and improvement. By creating a range of options, we increase the chances of finding an optimal prompt for the 8B model.

Generating the System Prompt

def generate_system_prompt(task, prompts):

completion = client.chat.completions.create(

model=MODEL_70B,

messages=[{

"role": "system",

"content": "Given a user-description of their task and a set of prompts, write a fantastic system prompt that describes the task to be done perfectly."

}, {

"role": "user",

"content": f"Task: {task}\nPrompts: {json.dumps(prompts)}"

}],

temperature=0.5,

max_tokens=1000,

)

return completion.choices[0].message.content.strip()

This function creates a system prompt that helps guide the 8B model in understanding and executing the task. It's a crucial step in ensuring that the smaller model can perform as effectively as possible.

Testing on the 8B Model

def test_8b(system_prompt, prompt):

completion = client.chat.completions.create(

model=MODEL_8B,

messages=[{

"role": "system",

"content": system_prompt

}, {

"role": "user",

"content": prompt

}],

temperature=0.5,

max_tokens=2000,

)

return completion.choices[0].message.content

This function tests the optimized prompts on the 8B model, allowing us to evaluate the effectiveness of our optimization process.

The User Interface: Simplifying Complexity

To make this tool accessible to a wide range of users, I've implemented a Gradio interface. This provides a user-friendly way to interact with the complex underlying processes:

with gr.Blocks() as iface:

gr.Markdown("# Llama 3.1 70B to 8B Prompt Conversion")

gr.Markdown("Convert prompts from Llama 3.1 70B to Llama 3.1 8B")

with gr.Row():

task_input = gr.Textbox(label="Task Description")

generate_button = gr.Button("Generate & Test Example Prompt")

with gr.Row():

prompt_input = gr.Textbox(label="Prompt Example", lines=5)

initial_response = gr.Textbox(label="70B Model Response", lines=5)

output = gr.Textbox(label="Conversion Output", lines=30)

groq_code_output = gr.Code(

label="Groq Code with 8B System Prompt and New Prompt",

language="python")

submit_button = gr.Button("Convert and Test")

# ... (click event handlers)

This interface allows users to input their task, generate an initial prompt, and then run the full optimization process with just a few clicks.

Why This Matters

This tool has significant implications for AI development and deployment:

1. Efficiency: By optimizing prompts for smaller models, we can reduce computational costs without sacrificing too much capability.

2. Accessibility: It makes advanced prompt engineering techniques available to a wider audience, fostering innovation.

3. Research: It provides a platform for studying the differences between large and small language models, potentially leading to new insights in AI development.

Try It Yourself

I've made this tool available as a Replit template, so you can try it out for yourself. Just add your Groq API key, and you're ready to go! Check it out here:

Conclusion

As we continue to push the boundaries of AI, tools like this will play a crucial role in making advanced technologies more accessible and efficient. I'm excited to see how the community will use and build upon this project. Whether you're optimizing AI deployments, studying model capabilities, or developing new prompt engineering strategies, I hope this tool will be a valuable resource in your AI journey.

Feel free to fork the project, contribute, or share your results. Let's work together to unlock the full potential of AI, making it more efficient and accessible for everyone!

The Spark of Inspiration

It all started when I was thinking about how to make this year's Fourth of July celebration a bit more special. As an indie developer always looking to push the boundaries, I thought, "What if we could have an AI-powered trivia game that never runs out of questions?" And thus, Infinite Fourth Trivia was born!

Leveraging Cutting-Edge AI

The heart of this project lies in its ability to generate an endless stream of USA-themed questions. To make this possible, I turned to two incredible technologies:

1. Groq's Lightning-Fast Inference: Groq's API allowed me to generate questions at breakneck speeds, ensuring a smooth user experience.

2. Google's Gemma 2 9b Model: This powerful language model provides the brains behind our questions, ensuring they're diverse, accurate, and engaging.

Here's a snippet of how I set up the Groq client in my FastAPI backend:

from groq import AsyncGroq
import os

client = AsyncGroq(api_key=os.environ.get("GROQ_API_KEY"))
GROQ_MODEL = "gemma2-9b-it"

Crafting the Perfect Question

One of the most crucial parts of the project was designing the prompt for our AI model. I wanted to ensure that questions were not only factual but also engaging and appropriate for a Fourth of July celebration. Here's a simplified version of the prompt I used:

trivia_prompt = f"""Generate a Fourth of July themed trivia multiple-choice question. The question should be {difficulty} difficulty, focusing on US independence, American Revolution, patriotic symbols, or Fourth of July traditions.

Format the response as a JSON object with the following structure:
{{
    "question": "The full text of the question",
    "options": ["A. First option", "B. Second option", "C. Third option", "D. Fourth option"],
    "correct_answer": "The letter of the correct answer (A, B, C, or D)",
    "explanation": "A brief explanation of the correct answer"
}}

Ensure that the question is factual, engaging, and appropriate for a Fourth of July celebration. Aim for a mix of historical facts, cultural traditions, and interesting tidbits about American independence and patriotism.

IMPORTANT: Do not repeat any of the following recent questions:
{recent_questions_str}

Reply only with the JSON object, no additional comments or explanations.
"""

Building a Responsive Frontend

For the frontend, I wanted something that was both visually appealing and easy to use. I opted for a combination of HTML, CSS (with Tailwind for quick styling), and JavaScript. Here's a small taste of the JavaScript that powers the game logic:

async function fetchAndDisplayQuestion(difficulty) {
    console.log('Fetching and displaying question with difficulty:', difficulty);
    try {
        const response = await axios.post('/fourth_trivia', { difficulty }, {
            headers: {
                'Content-Type': 'application/json'
            }
        });
        console.log('Response received:', response.data);
        if (response.data.error) {
            console.error('Error from server:', response.data.error);
            displayError("Sorry, there was an error fetching the question. Please try again.");
        } else {
            displayQuestion(response.data);
            prefetchNextQuestion(difficulty);  // Prefetch the next question after displaying current one
        }
    } catch (error) {
        console.error('Error fetching question:', error);
        displayError("Sorry, there was an error connecting to the server. Please check your internet connection and try again.");
    }
}

Challenges and Learnings

Building Infinite Fourth Trivia wasn't without its challenges. Ensuring question uniqueness, handling various edge cases, and optimizing for performance were all hurdles I had to overcome. But with each challenge came a learning opportunity, and I'm excited to apply these lessons to future projects.

The Road Ahead

This is just the beginning for Infinite Fourth Trivia. I have plans to expand the game with more features, such as multiplayer modes, themed question sets, and perhaps even voice-activated gameplay. The possibilities are as endless as the questions themselves!

Try It Out!

I'm thrilled to share Infinite Fourth Trivia with all of you. Whether you're a history buff, a casual trivia fan, or just looking for a fun way to celebrate the Fourth, I hope you'll give it a try. You can find the game at https://infinite4thtrivia.replit.app, and the source code is available on Replit.

Happy Fourth of July, and happy trivia-ing!

The Spark of Inspiration

The core idea behind Chain-of-Table is brilliantly simple yet powerful: instead of relying solely on textual reasoning chains, why not leverage the structured nature of tables themselves to represent intermediate thoughts? This approach allows large language models (LLMs) to dynamically plan and execute a series of table operations, effectively "evolving" the table to better answer complex questions.

Excited by the potential, I fired up Replit and got to work. Here's a glimpse into how I implemented the key components:

async def chain_of_table_reasoning(df, question):
    messages = [{
        "role": "system",
        "content": system_prompt
    }, {
        "role": "user",
        "content": f"Table:\n{df.to_string()}\n\nQuestion: {question}"
    }]

    tools = [{
        "type": "function",
        "function": {
            "name": func_name,
            "description": f"Perform {func_name} operation on the table",
            "parameters": {
                "type": "object",
                "properties": {
                    "args": {
                        "type": "object",
                        "description": f"Arguments for {func_name} operation"
                    }
                },
                "required": ["args"]
            }
        }
    } for func_name in [
        "add_col", "select_row", "select_col", "group_by", "sort_by",
        "get_table_info"
    ]]

    # ... (loop for iterative reasoning)

This function sets up the core reasoning loop, allowing the LLM to iteratively transform the table using a set of predefined operations.

Supercharging with Groq

To really push the boundaries of performance, I decided to leverage Groq's lightning-fast inference API. Their function calling feature turned out to be a game-changer for this project. The speed is truly remarkable, making the Chain-of-Table reasoning feel almost real-time.

Here's a snippet showing how I integrated Groq:

client = AsyncGroq(api_key=os.environ.get("GROQ_API_KEY"))

MODEL = "llama3-70b-8192"

# ... (in the reasoning loop)
response = await client.chat.completions.create(
    model=MODEL,
    messages=messages,
    tools=tools,
    tool_choice="auto",
)

Bringing It All Together with Gradio

To make this project accessible and interactive, I built a simple Gradio interface. This allows users to input their own tables and questions, seeing the Chain-of-Table reasoning in action. Here's a glimpse of the Gradio setup:

with gr.Blocks() as demo:
    gr.Markdown("# Chain-of-Table Reasoning")

    with gr.Row():
        with gr.Column():
            table_input = gr.TextArea(label="Input Table (CSV format)")
            upload_button = gr.UploadButton("Upload CSV", file_types=["csv"])
            upload_button.upload(lambda file: file.decode('utf-8'),
                                 upload_button, table_input)

        with gr.Column():
            question_input = gr.TextArea(label="Question")
            submit_button = gr.Button("Submit")

    output = gr.TextArea(label="Answer")

    submit_button.click(process_question,
                        inputs=[table_input, question_input],
                        outputs=output)

    gr.Examples(examples=examples, inputs=[table_input, question_input])

The Power of Open Development

One of the most exciting aspects of this project has been the ability to rapidly prototype and deploy using Replit. Their platform is truly a godsend for going from idea to MVP at lightning speed. Plus, deployment is a breeze!

What's Next?

I'm thrilled to see what the community builds with this Chain-of-Table implementation. The combination of structured table reasoning, Groq's speed, and Replit's accessibility creates a powerful foundation for innovation.

I've made a template available so you can easily get started with your own experiments. Who knows? Your next AI breakthrough might be just a few table operations away!

Remember, the world of AI moves fast, and sometimes the most exciting developments happen when we combine cutting-edge research with practical implementation. So grab that template, fire up your Replit, and let's see what amazing things we can build together!

The Concept: Why Settle for One AI When You Can Have Four?

The idea behind this project is simple yet powerful: instead of relying on a single AI model to answer user queries, why not leverage the strengths of multiple models? This approach allows us to tap into the diverse knowledge and capabilities of different AI architectures, potentially leading to more comprehensive and nuanced responses.

Here's how it works:

1. We take the user's question and send it to four different language models.

2. Each model processes the query and generates its own response.

3. A fifth model, acting as an "aggregator," synthesizes these responses into a single, coherent answer.

The result? A chatbot that provides multi-perspective, well-rounded answers to user queries.

The Tech Stack: Groq, Replit, and Gradio

For this project, I've leveraged some cutting-edge tools:

• Groq: This incredible inference API allows us to run multiple AI models with lightning-fast speed. It's the secret sauce that makes real-time processing of multiple models feasible.

• Replit: I've set up the project as a Replit template, making it super easy for anyone to clone and run their own instance of the chatbot.

• Gradio: This library provides a simple yet powerful interface for our chatbot, allowing users to interact with it through a clean, web-based UI.

The Code: Let's Break It Down

Let's walk through some key parts of the code to understand how this MoA chatbot works.

Setting Up the Models

reference_models = [
    "gemma2-9b-it", "llama3-70b-8192", "llama3-8b-8192", "mixtral-8x7b-32768"
]

aggregator_model = "llama3-70b-8192"

Here, we define our four reference models and the aggregator model. I've chosen a mix of different architectures and sizes to get a diverse range of responses.

The Aggregator's System Prompt

The aggregator model needs to know its job. We provide it with a detailed system prompt:

aggregator_system_prompt = """As an advanced language model, your task is to synthesize responses from multiple AI models into a single, coherent, and high-quality answer. Follow these guidelines:

1. Analyze and compare: Carefully examine the responses from different models, noting similarities, differences, and unique insights.
2. Evaluate accuracy: Critically assess the information provided, identifying and discarding any inaccuracies or biases.
...
"""

This prompt is crucial as it guides the aggregator in combining the various model outputs effectively.

Running the Models

The run_llm function handles individual model calls:

async def run_llm(model, message, history):
    messages = [{
        "role": "user" if i % 2 == 0 else "assistant",
        "content": str(msg)
    } for i, msg in enumerate(history + [message])]

    response = await client.chat.completions.create(
        model=model,
        messages=messages,
        temperature=0.7,
        max_tokens=512,
    )
    return response.choices[0].message.content

We use async calls to Groq's API for optimal performance.

The Main Chat Function

The heart of our chatbot is the chat_with_mixture_of_agents function:

async def chat_with_mixture_of_agents(message, history):
    # Run reference models
    tasks = [run_llm(model, message, history) for model in reference_models]
    results = await asyncio.gather(*tasks)

    # Prepare aggregator input
    aggregator_input = "\n\n".join([
        f"{model}:\n{response}"
        for model, response in zip(reference_models, results)
    ])

    # Run aggregator model
    aggregator_messages = [
        {
            "role": "system",
            "content": aggregator_system_prompt
        },
        {
            "role": "user",
            "content": f"User query: {message}\n\n{aggregator_input}"
        },
    ]

    response_content = ""
    stream = await client.chat.completions.create(
        model=aggregator_model,
        messages=aggregator_messages,
        temperature=0.7,
        max_tokens=1024,
        top_p=1,
        stop=None,
        stream=True,
    )

    async for chunk in stream:
        content = chunk.choices[0].delta.content
        if content:
            response_content += content
        yield response_content

This function orchestrates the entire process:

1. It runs all reference models concurrently.

2. It prepares the input for the aggregator model.

3. It streams the aggregator's response back to the user in real-time.

The User Interface

We use Gradio to create a simple chat interface:

with gr.Blocks(fill_height=True, head=js) as demo:
    gr.ChatInterface(
        chat_with_mixture_of_agents,
        clear_btn=None,
        undo_btn=None,
        retry_btn=None,
        fill_height=True,
        examples=[
            "Explain the concept of quantum entanglement in simple terms.",
            "What are the potential implications of artificial general intelligence on society?",
            # ... more example questions ...
        ])

This creates a clean, user-friendly interface with some example questions to get users started.

Conclusion and Future Directions

This MoA chatbot is just the beginning. There are so many exciting directions we could take this:

• Experimenting with different combinations of models

• Implementing more sophisticated aggregation techniques

• Adding user controls for model selection or weighting

• Integrating external knowledge sources

The possibilities are endless, and I can't wait to see what the community does with this concept. Remember, the project is open-source, so feel free to fork it, modify it, and make it your own!

If you want to try it out yourself, head over to the Replit template and start chatting with our AI panel. And don't forget to let me know what you think or if you have any cool ideas for improvements.

Happy coding, and may your AI adventures be ever exciting! 🚀🤖

Let's dive into the nitty-gritty of this AI-powered polyglot helper!

The Tech Stack 🛠️

First things first, let's talk about the awesome tech that makes this possible:

• Groq API: The powerhouse behind our natural language processing

• Gradio: For that slick, user-friendly interface

• Replit: Our trusty hosting platform

• Python: The glue that holds it all together

Key Features 🌟

1. Multilingual Support: We're talking 9 languages here! From English to Portuguese, we've got you covered.

2. Real-time Speech Recognition: Thanks to Groq's Whisper API, we can turn your speech into text in a snap.

3. Intelligent Feedback: Grammar, vocabulary, pronunciation - we analyze it all!

4. Contextual Responses: Our AI doesn't just respond; it converses naturally.

5. Instant Translations: Because sometimes you need that extra help to understand.

The Code Breakdown 💻

Let's look at some of the cool parts of our code:

Speech Recognition

We're using Groq's Whisper API for transcription. Check out this async function:

async def transcribe_audio(audio_file):
    try:
        form_data = FormData()
        form_data.add_field('file', audio_file, filename='audio.wav', content_type='audio/wav')
        form_data.add_field('model', 'whisper-large-v3')
        # ... more fields ...

        async with aiohttp.ClientSession() as session:
            async with session.post("https://api.groq.com/openai/v1/audio/transcriptions",
                                    headers={"Authorization": f"Bearer {GROQ_API_KEY}"},
                                    data=form_data) as response:
                # ... handle response ...
                return result["text"]
    except ClientError as e:
        raise GroqError(f"Network error during transcription: {str(e)}")

This function handles the audio file, sends it to Groq's API, and returns the transcribed text. Pretty neat, huh?

Language Analysis

Once we have the text, we analyze it:

async def analyze_language(text, target_language, history):
    prompt = f"""Analyze the following text in {target_language}, considering the conversation history:
    # ... prompt details ...
    """

    response = await async_groq_client.chat.completions.create(
        model="llama3-70b-8192",
        messages=[{"role": "system", "content": "You are an expert language tutor."},
                  {"role": "user", "content": prompt}],
        temperature=0.3,
        max_tokens=1000,
        response_format={"type": "json_object"})
    return json.loads(response.choices[0].message.content)

This function crafts a prompt for Groq's language model, asking it to analyze grammar, vocabulary, and more. We get back a structured JSON response with all the juicy details.

Conversation Management

To keep things flowing naturally, we maintain a conversation history:

def update_conversation_history(user_id, target_language, new_entry):
    key = f"{user_id}_{target_language}"
    history = get_conversation_history(user_id, target_language)
    history.append(new_entry)
    history = history[-5:]  # Keep only the last 5 interactions
    db[key] = history

This function updates our conversation history, keeping the last 5 interactions for context. It's like giving our AI a short-term memory!

The User Experience 🎨

Using Gradio, we've created an interface that's both powerful and easy to use:

iface = gr.Interface(
    fn=language_tutor,
    inputs=[
        gr.Audio(type="filepath", label="Speak here"),
        gr.Dropdown(["English", "Chinese", "German", "Spanish", "Russian", "Korean",
                     "French", "Japanese", "Portuguese"],
                    label="Select target language"),
        # ... more inputs ...
    ],
    outputs=[
        gr.Textbox(label="Session ID"),
        gr.Textbox(label="Target Language"),
        gr.Textbox(label="Transcription"),
        gr.Textbox(label="Language Analysis"),
        gr.Textbox(label="AI Response"),
        gr.Textbox(label="Translation"),
        gr.Textbox(label="Session Management Message")
    ],
    title="AI Language Learning Assistant",
    description="Speak in your target language and get instant feedback and responses!"
)

This creates a web interface where users can speak, select their language, and get instant feedback. It's like having a language tutor in your browser! 🌐👩‍🏫

Wrapping Up 🎁

And there you have it, folks! A full-fledged AI language tutor that you can spin up with just a few clicks on Replit. It's been a blast putting this together, and I hope it inspires you to create your own AI-powered tools.

Remember, you don't need to be at a fancy hackathon to innovate. Sometimes, the best ideas come when you're #Hack@Home! 🏠💡

So, what are you waiting for? Fork the Replit template, add your Groq API key, and start your polyglot journey today!

Happy coding, and may your conversations be ever multilingual! 🌍🗣️

The Inspiration 💡

So, Karpathy tweeted about wanting a built-in feature in LLM interfaces that provides proof for the information they give. You know how it is - we often find ourselves manually verifying things like command switches or API details. Well, that got my gears turning, and I thought, "Why not create something that does this automatically?" 🤔

The Solution: Fact-Checked LLM Chat Interface 🧠🔍

I threw together a quick and dirty (but oh-so-cool) Fact-Checked LLM Chat interface. Here's how it works:

1. We use Anthropic's Claude 3.5 Sonnet to generate responses.

2. The response is streamed back to the user in real-time.

3. We then use Brave Search to find relevant sources for fact-checking.

4. The content from these sources is fed back to Claude to verify its original claims.

5. Finally, we present the user with the fact-check results and the source URL.

Let's break down some of the key components of this code, shall we? 🛠️

Setting Up the Environment

First things first, we import all the necessary libraries and set up our API keys:

import os
import asyncio
import httpx
import gradio as gr
from anthropic import AsyncAnthropic
import json
import re

async_client = AsyncAnthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
BRAVE_API_KEY = os.environ.get("BRAVE_API_KEY")
CLAUDE_MODEL = "claude-3-5-sonnet-20240620"

Fetching Search Results

We use Brave Search API to find relevant sources:

async def fetch_brave_search_results(query: str, count: int = 5) -> dict:
    url = "https://api.search.brave.com/res/v1/web/search"
    headers = {
        "Accept": "application/json",
        "Accept-Encoding": "gzip",
        "X-Subscription-Token": BRAVE_API_KEY
    }
    params = {
        "q": query,
        "count": count,
        "country": "us",
        "search_lang": "en",
    }

    async with httpx.AsyncClient() as client:
        response = await client.get(url, headers=headers, params=params)
        response.raise_for_status()
        return response.json()

Streaming Claude's Responses

We stream Claude's responses for a snappy user experience:

async def ask_claude_streaming(message):
    completion = async_client.messages.stream(
        max_tokens=4096,
        messages=[{
            "role": "user",
            "content": message
        }],
        model=CLAUDE_MODEL,
    )
    async with completion as stream:
        async for text in stream.text_stream:
            yield text

Fact-Checking Magic ✨

The real magic happens in the fact_check function. We fetch content from the search results, have Claude analyze it, and return the fact-check results:

async def fact_check(llm_response: str, query: str, search_results: dict,
                     first_page_content: str) -> str:
    # ... (check the full code for details)
    fact_check_prompt = f"""
    Given the following claim and markdown content from a webpage, determine if the claim is supported by the content. 
    Respond with a JSON object containing two fields: "claim_supported" (boolean) and "supporting_facts" (string).
    Ensure the "supporting_facts" field is a single line string with no line breaks.

    Claim: {llm_response}

    Query: {query}

    Markdown Content:
    {markdown_content}

    JSON Response:
    """

    fact_check_result = ''.join(
        [chunk async for chunk in ask_claude_streaming(fact_check_prompt)])
    # ... (process the result)

Putting It All Together

The chat_with_fact_check function orchestrates the whole process:

async def chat_with_fact_check(message, history):
    # ... (see full code for details)
    llm_response = ""
    async for chunk in ask_claude_streaming(json.dumps(messages)):
        llm_response += chunk
        yield history + [[message, llm_response]]

    # Fact-checking after the full response
    fact_check_result = await fact_check(llm_response, message, search_results,
                                         first_page_content)

    yield history + [[message, llm_response],
                     ["Fact Check", fact_check_result]]

The Result 🎉

What we end up with is a chat interface that not only provides AI-generated responses but also fact-checks them on the fly! It's like having a super-smart assistant with a built-in fact-checker. How cool is that? 😎

Try It Yourself!

I've made a Replit template so you can fork, build, and improve upon this project. Feel free to play around with it and see what awesome features you can add!

Remember, this is just a quick demo, but the possibilities are endless. You could extend this to support more sources, improve the fact-checking algorithm, or even add visual elements to make the proof more engaging.

So, what do you think? Ready to take your LLM interactions to the next level with some automatic fact-checking? Let me know what cool modifications you come up with!

Happy coding, and may your AI assistants always be truthful! 🤖✅

The Backend Magic 🧙‍♂️

Let's start by looking at the heart of our application - the FastAPI backend. Here's where the real magic happens:

import os
import io
import logging
from fastapi import FastAPI, File, UploadFile, Request, WebSocket
from fastapi.responses import HTMLResponse
from fastapi.templating import Jinja2Templates
from groq import AsyncGroq

app = FastAPI()
templates = Jinja2Templates(directory="templates")

GROQ_API_KEY = os.environ.get("GROQ_API_KEY")
groq_client = AsyncGroq(api_key=GROQ_API_KEY)

We're using FastAPI for its speed and ease of use, and we're integrating with Groq's AsyncGroq client to tap into their lightning-fast Whisper v3 model.

The star of the show is our /transcribe endpoint:

@app.post("/transcribe")
async def transcribe_audio(file: UploadFile = File(...)):
    contents = await file.read()
    audio_file = io.BytesIO(contents)
    audio_file.name = "audio.wav"

    try:
        transcription = await groq_client.audio.transcriptions.create(
            file=audio_file,
            model="whisper-large-v3",
            response_format="verbose_json"
        )
        return {"transcription": transcription.text}
    except Exception as e:
        return {"error": f"Error during transcription: {str(e)}"}

This endpoint receives audio chunks, processes them using Groq's Whisper v3 model, and returns the transcription. It's asynchronous, which means it can handle multiple requests efficiently without blocking.

The Frontend Experience 🎨

On the frontend, we're keeping things simple yet effective with a straightforward HTML/JS setup:

<button id="startButton">Start Recording</button>
<button id="stopButton" disabled>Stop Recording</button>
<div id="transcription"></div>
<div id="status"></div>

The real action happens in our JavaScript:

startButton.onclick = async () => {
    try {
        const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
        mediaRecorder = new MediaRecorder(stream);

        mediaRecorder.ondataavailable = (event) => {
            audioChunks.push(event.data);
            if (audioChunks.length === 5) {  // Send every 5 chunks
                sendAudioChunks();
            }
        };

        mediaRecorder.start(1000);  // Capture in 1-second intervals
        // ... (update UI)
    } catch (error) {
        updateStatus(`Error starting recording: ${error.message}`);
    }
};

We're using the MediaRecorder API to capture audio in 1-second chunks. Every 5 chunks, we send the data to our backend for transcription.

The Secret Sauce: Groq's Whisper v3 🚀

The real power behind this application is Groq's Whisper v3 model. It's incredibly fast and accurate, allowing us to transcribe audio in near real-time. By leveraging Groq's API, we're able to offload the heavy lifting of transcription to their specialized hardware, resulting in lightning-fast results.

Putting It All Together 🏗️

When a user starts recording, we capture audio chunks and send them to our backend. The backend then uses Groq's Whisper v3 model to transcribe the audio and sends the results back to the frontend, where it's displayed in real-time.

This architecture allows for a smooth, responsive user experience while taking advantage of the incredible speed of Groq's technology.

Wrapping Up 🎁

And there you have it! A real-time audio transcription service that leverages the power of Groq's Whisper v3 model. This project demonstrates how we can combine modern web technologies with cutting-edge AI to create powerful, user-friendly applications.

Feel free to clone the repo, experiment with it, and let me know what cool features you add! Happy coding, everyone! 🚀👨‍💻👩‍💻

1. FastAPI Framework

We're using FastAPI as our web framework. It's fast, easy to use, and perfect for building APIs. Here's how we set up our basic app:

from fastapi import FastAPI
import uvicorn

app = FastAPI()

if __name__ == "__main__":
    uvicorn.run(app, port=8080, host="0.0.0.0")

2. Message Completion Prediction

One of the coolest features of this chatbot is its ability to predict message completions. We use Groq's API to assess the likelihood that a user's message is complete:

response = await client.chat.completions.create(
    model=MODEL_TO_USE,
    messages=[{
        "role": "system",
        "content": "It is extremely important that you are brief and that you only respond with a number between 0-100 to show likelihood that the user message is complete and can be responded to..."
    }, {
        "role": "user",
        "content": f"{text}"
    }],
)

If the likelihood is below 90%, we generate a predicted ending for the message.

3. Similarity Detection

To avoid redundant processing, we implement a similarity check using embeddings:

async def check_if_same(combined_message):
    global last_message_embedding, last_response
    embedding_response = await embed_client.embeddings.create(
        model=EMBEDDING_MODEL_TO_USE, input=combined_message)
    current_embedding = np.array(embedding_response.data[0].embedding)

    if last_message_embedding is not None:
        similarity = 1 - cosine(last_message_embedding, current_embedding)
        if similarity > 0.9:
            return True, similarity, last_response

    last_message_embedding = current_embedding
    return False, 0, None

This function helps us identify if the current message is very similar to the previous one, allowing us to reuse the last response if appropriate.

4. Response Generation

If the message is unique or complete, we generate a new response using Groq's API:

response = await client.chat.completions.create(
    model=MODEL_TO_USE,
    messages=[{
        "role": "system",
        "content": "Generate a response to the user message"
    }, {
        "role": "user",
        "content": combined_message
    }],
)

5. User Interface

We've created a simple yet effective UI using HTML, Bootstrap, and jQuery. The interface allows users to input their messages and displays the AI's responses in real-time.

<div class="container text-center">
    <h2>Enter Your Message</h2>
    <div class="d-flex justify-content-center">
        <input type="text" class="inputBox" id="userInput">
        <div class="spinner-border text-primary d-none" id="loadingSpinner" role="status">
            <span class="sr-only">Loading...</span>
        </div>
    </div>
    <div class="response-area" id="responseArea"></div>
</div>

The JavaScript code handles user input and updates the UI with the AI's responses:

$('#userInput').on('keyup', function(event) {
    if (event.key === ' ' || event.key === '.' || event.key === '?' || event.key === '!') {
        // Send request to server
        // ...
    }
});

This project demonstrates how we can combine multiple AI models and fast inference engines like Groq to create a more intelligent and responsive chatbot. The system's ability to predict message completions, detect similarities, and generate context-aware responses makes for a much more natural conversation flow.

I'm constantly working on improving this system, and I'd love to hear your thoughts or suggestions. Feel free to try it out and let me know what you think!