Package Exports

@darkbing/knowledge-retrieval
@darkbing/knowledge-retrieval/dist/index.js

This package does not declare an exports field, so the exports above have been automatically detected and optimized by JSPM instead. If any package subpath is missing, it is recommended to post an issue to the original package (@darkbing/knowledge-retrieval) to support the "exports" field. If that is not possible, create a JSPM override to customize the exports field for this package.

Readme

@lodi-know/knowledge-retrieval

A powerful web crawler and knowledge processing toolkit for extracting and managing web content with AI-powered capabilities.

🚀 Features

🌐 Intelligent Web Crawling
📄 Multi-format Content Processing
🤖 AI-Enhanced Knowledge Extraction
🧹 Robust Resource Management

📦 Installation

npm install @lodi-know/knowledge-retrieval

💻 Quick Start

CLI Usage

# Interactive CLI
npx knowledge-retrieval

# Crawl a website
npx knowledge-retrieval crawl https://example.com

# Process crawled resources
npx knowledge-retrieval process

Programmatic Usage

import { 
  Crawler, 
  KnowledgeProcessor, 
  ResourceCleaner 
} from '@lodi-know/knowledge-retrieval';

// Crawl a website
const crawler = new Crawler();
await crawler.crawl('https://example.com');

// Process crawled content
const processor = new KnowledgeProcessor();
await processor.processResources();

🛠️ Configuration

Create a .env file with the following options:

MAX_CRAWL_DEPTH=3
MAX_CRAWL_PAGES=50
REQUEST_TIMEOUT=5000

🔧 Commands

npm run crawl: Start web crawling
npm run process: Process crawled resources
npm run cleanup: Clean up crawled and processed data

🤝 Contributing

Fork the repository
Create your feature branch
Commit your changes
Push to the branch
Create a new Pull Request

📄 License

MIT License

@darkbing/knowledge-retrieval