https://github.com/defrecord/llm-lab

LLM Tools Lab: Testing environment for exploring LLM CLI tools (llm (0.21), ttok, strip-tags) with multiple providers (Ollama, Claude, Gemini, Bedrock, and OpenAI).
https://github.com/defrecord/llm-lab

Last synced: about 1 year ago
JSON representation

LLM Tools Lab: Testing environment for exploring LLM CLI tools (llm (0.21), ttok, strip-tags) with multiple providers (Ollama, Claude, Gemini, Bedrock, and OpenAI).

• Host: GitHub
• URL: https://github.com/defrecord/llm-lab
• Owner: defrecord
• License: apache-2.0
• Created: 2025-01-31T15:59:38.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2025-03-19T14:50:25.000Z (about 1 year ago)
• Last Synced: 2025-03-19T15:40:02.396Z (about 1 year ago)
• Language: Shell
• Homepage:
• Size: 420 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 23
• Metadata Files:
  • Readme: README.org
  • Contributing: CONTRIBUTING.org
  • License: LICENSE
  • Code of conduct: CODE_OF_CONDUCT.md
  • Security: SECURITY.md

Awesome Lists containing this project

README

          
#+TITLE: LLM Lab

#+STARTUP: showeverything

* Overview

A hands-on learning environment for exploring LLM CLI tools with practical examples and exercises.

** Key Resources

- [[https://llm.datasette.io/en/stable/help.html][LLM CLI Documentation]]

- [[https://github.com/simonw/ttok][ttok]] - Token counting tool

- [[https://github.com/simonw/strip-tags][strip-tags]] - HTML cleanup

- [[https://github.com/simonw/files-to-prompt][files-to-prompt]] - File content extraction

* Getting Started

** Quick Start

1. Clone the repository

2. Run ~make init~ to set up your environment

3. Run ~make check-env~ to verify your setup

4. Run ~make essential-examples~ to process introductory examples

5. See examples directory to begin learning

** Available Models

- Claude 3 models via llm-claude-3

- Google's Gemini models

- Local models via Ollama

- AWS Bedrock integration

* Learning Path

All examples are self-contained and build upon each other:

** Essential Examples

These are the core examples to get started:

1. [[file:examples/00-getting-started.org][Getting Started]] - Basic LLM usage and setup

   - First commands and responses

   - Understanding the environment

   - Working with models

   - Practice: Basic prompts and responses

2. [[file:examples/01-templates.org][Templates]] - Working with system prompts

   - Creating custom templates

   - Using built-in templates

   - Template best practices

   - Practice: Create and use custom templates

** Advanced Topics

After completing the essential examples, explore these advanced topics:

1. [[file:examples/03-agents.org][Agents]] - Specialized roles and interactions

   - Agent types and purposes

   - Multi-agent conversations

   - Agent collaboration patterns

   - Template best practices

2. [[file:examples/02-context-management.org][Context Management]] - Managing conversations

   - Maintaining context

   - Structured interactions

   - Memory handling

3. [[file:examples/04-embeddings-intro.org][Embeddings Introduction]] - Vector representations

   - Understanding embeddings

   - Basic vector operations

   - Similarity searches

4. [[file:examples/05-photo-embeddings.org][Photo Embeddings]] - Working with images

   - Image analysis

   - Semantic search

   - Visual relationships

5. [[file:examples/06-advanced-usage.org][Advanced Usage]] - Complex workflows

   - Integration patterns

   - Custom solutions

   - Best practices

6. [[file:examples/50-ollama-models.org][Ollama Models]] - Local model usage

   - Setting up Ollama

   - Model management

   - Performance considerations

* Project Structure

The workspace is organized for easy navigation:

- ~examples/~ :: Step-by-step learning materials

  - ~51-sqlite-queries.org~ :: SQLite analytics for LLM logs

  - ~src/~ :: Tangled code from examples

- ~templates/~ :: Analysis templates and frameworks

  - ~sin-framework.md~ :: System analysis framework

  - ~sin-execution-plan.md~ :: Implementation planning

  - ~sin-execute-and-document.md~ :: Results documentation

- ~src/sql/~ :: Organized SQL queries for analysis

  - ~advanced/~ :: Complex analytics queries

  - ~basic/~ :: Basic usage statistics

  - ~cost/~ :: Token cost analysis

  - ~usage/~ :: Usage pattern analysis

- ~scripts/~ :: Utility scripts

  - ~register-sin.sh~ :: SIN template registration

- ~prompts/~ :: Example system prompts

- ~docs/~ :: Additional guides and references

- ~data/~ :: Your working directory for outputs

** SQLite Analytics

Comprehensive SQLite queries for analyzing LLM usage logs:

*** Basic Analytics

- Conversation counts and trends

- Model usage statistics

- Temporal analysis

*** Advanced Analytics

- Response time analysis

- Token usage patterns

- Full-text search capabilities

*** Cost Analysis

- Token usage tracking

- Cost estimation by model

- Usage optimization insights

** SIN Templates

The Structured Intelligence Network (SIN) provides a systematic approach to LLM analysis:

*** Framework Analysis

- Analysis categories and metrics

- Data collection methods

- Evaluation criteria

- Implementation steps

- Reporting structure

*** Execution Planning

- Implementation schedule

- Data collection plan

- Analysis procedures

- Resource allocation

- Risk management

*** Documentation

- Executive summary

- Analysis results

- Technical details

- Recommendations

- Next steps

* Need Help?

- Check the example documentation

- Review the LLM CLI docs

- See CONTRIBUTING.org for development details

* References

- [[https://llm.datasette.io/][LLM CLI Documentation]]

- [[https://simonwillison.net/2023/May/18/cli-tools-for-llms/][Introduction to LLM CLI Tools]]