Skip to content

2. Epic News Tools Handbook

This handbook provides a comprehensive overview of all tools available to agents within the epic_news project, including native CrewAI tools, custom-built tools, and integrations from the Composio library.

1. Tool Integration & Best Practices

1.1. Modular Architecture & Factories

  • Factory Pattern: Use factory functions to organize and provide tools. This allows for centralized management and easy modification.
  • Domain Separation: Group tools by functionality into separate files (e.g., web_tools.py, finance_tools.py, github_tools.py).
  • Graceful Degradation: If a tool fails to initialize (e.g., missing API key), the factory should handle it gracefully (e.g., return an empty list) so the application can proceed without it.
# Example: src/epic_news/tools/web_tools.py
def get_search_tools():
    return [PerplexitySearchTool()]

# Example: graceful degradation (src/epic_news/tools/github_tools.py)
def get_github_tools() -> list[BaseTool]:
    if not os.getenv("GITHUB_TOKEN"):
        return []  # tool unavailable without a token
    return [GitHubSearchTool()]

1.2. API Key Management

  • Security: All API keys and sensitive credentials must be stored in environment variables (e.g., in a .env file) and never hardcoded in source code.
  • Validation: Factory functions should validate the presence of required environment variables.
  • Naming Convention: Use a consistent naming convention, such as TOOL_NAME_API_KEY.
# .env file
TAVILY_API_KEY=tvly-xxxxxx
SERPER_API_KEY=xxxxxx
FIRECRAWL_API_KEY=fc-xxxxxx
RAPIDAPI_KEY=rapi-xxxxxx

1.3. Performance & Testing

  • Asynchronous Execution: Enable async for independent, I/O-bound tasks.
  • Caching: Implement caching for expensive or frequently repeated API calls.
  • Testing: Write unit tests for tool factories and integration tests for tool combinations. Mock external APIs to avoid costs and dependencies during testing.

1.3.1. Testing Policy (PR-001)

  • JSON outputs only: Tool _run() results MUST be JSON strings. See src/epic_news/tools/_json_utils.py::ensure_json_str().
  • No live network in tests: mock external APIs with stubs.
  • Run tests:
uv run pytest -q tests/tools/test_json_outputs.py tests/tools/test_json_contract_ratchet.py
  • See also README section “Testing” for full-suite commands and linting.

1.4. Scraper Factory (PR-002)

  • What it does: Centralizes selection of the web scraper provider via src/epic_news/tools/scraper_factory.py::get_scraper().
  • Default: ScrapeNinjaTool.
  • Switching providers: Set WEB_SCRAPER_PROVIDER in your .env to scrapeninja (default) or firecrawl.
  • Required keys:
  • RAPIDAPI_KEY for ScrapeNinja
  • FIRECRAWL_API_KEY for Firecrawl

Usage example:

from epic_news.tools.scraper_factory import get_scraper

scraper = get_scraper()  # selected by WEB_SCRAPER_PROVIDER
result_json = scraper.run({"url": "https://example.com"})

2. Available Tools by Category

2.1. 🌐 Web & Search Tools

Tool Source Description API Key Required
TavilyTool Composio/Custom Advanced search with domain filtering. TAVILY_API_KEY
SerperApiTool CrewAI Google search via Serper.dev for general web searches. SERPER_API_KEY
FirecrawlScrapeWebsiteTool CrewAI/Custom Extracts full content from a specific URL. FIRECRAWL_API_KEY
FirecrawlSearchTool Custom Performs a targeted search within a specific website. FIRECRAWL_API_KEY
ScrapeNinjaTool Custom Advanced web scraping with proxy and JS extraction. RAPIDAPI_KEY
CappedScrapeWebsiteTool CrewAI/Custom Backup scraper (crewai_tools ScrapeWebsiteTool) returning JSON capped at SCRAPE_MAX_CHARS characters (default 12000). No
RecentSearchTool In-repo recent_search: Perplexity web search limited to the last 12 months (search_recency="year"), JSON result; on failure returns a JSON error telling the agent to use hybrid_search. Used by PESTEL. PERPLEXITY_API_KEY
EXASearchTool CrewAI Specialized search with similarity functions. EXA_API_KEY
BrowserbaseLoadTool CrewAI Browser-based content loading for dynamic sites. BROWSERBASE_API_KEY
YoutubeVideoSearchTool CrewAI Finds relevant video content on YouTube. YouTube API Key
PerplexityAI Composio Natural language search with sophisticated AI models. PERPLEXITY_API_KEY
SeleniumScrapingTool CrewAI Browser automation for JS-heavy sites. (ScrapeNinja is preferred). No

2.2. 📄 Document & File Tools

Tool Source Description API Key Required
OutputFileReadTool In-repo Reads a text file inside its root only (default output/); use it instead of FileReadTool on agents that browse the web. No
FileReadTool CrewAI Reads any local file (unscoped). Never on an agent that also searches or scrapes. No
FileWriteTool CrewAI Writes content to a specified file. No
DirectorySearchTool CrewAI Searches for files within a directory. No
DirectoryReadTool CrewAI Reads the contents of a directory. No
PDFSearchTool CrewAI Extracts and searches content from PDF files. No
DOCXSearchTool CrewAI Searches content in Microsoft Word documents. No
CSVSearchTool CrewAI Searches and analyzes data in CSV files. No
JSONSearchTool CrewAI Searches and manipulates data in JSON files. No
XMLSearchTool CrewAI Processes and searches XML documents. No
ExcelSearchTool CrewAI Analyzes data in Excel spreadsheets. No
MDXSearchTool CrewAI Processes and searches MDX files. No

2.3. 📊 Data & Analytics Tools

Tool Source Description API Key Required
YahooFinanceNewsTool Custom Fetches the latest financial news for a specific ticker. No
YahooFinanceHistoryTool Custom Gets historical price data for a ticker. No
YahooFinanceCompanyInfoTool Custom Gets detailed company information (summary, industry). No
AlphaVantageOverviewTool Custom Fetches fundamental company data (P/E, EPS). ALPHA_VANTAGE_API_KEY
CoinMarketCapInfoTool Custom Gets detailed info for a cryptocurrency. COINMARKETCAP_API_KEY
KrakenTickerInfoTool Custom Fetches real-time ticker data from Kraken exchange. No
TechStackAnalysisTool Custom Identifies the technology stack of a website. SERPER_API_KEY
GoogleFactCheckTool Custom Searches for fact-checked claims via Google's API. GOOGLE_API_KEY

2.4. 💻 Code & Development Tools

Tool Source Description API Key Required
GithubSearchTool CrewAI Searches GitHub repositories and code. GITHUB_TOKEN
CodeDocsSearchTool CrewAI Searches code documentation. No
CodeInterpreterTool CrewAI Executes and interprets Python code. No
ShellTool Composio Executes shell commands. No

2.5. 🤖 AI & RAG Tools

Tool Source Description API Key Required
RagTool CrewAI/Composio Generic Retrieval-Augmented Generation capabilities. Varies
DallETool CrewAI Generates images using DALL-E. OPENAI_API_KEY
VisionTool CrewAI Provides computer vision and image analysis. OPENAI_API_KEY

2.6. 🌍 Specialized & Integration Tools

Tool Source Description API Key Required
AccuWeatherTool Custom Gets current weather conditions for a location. ACCUWEATHER_API_KEY
AirtableTool Custom Creates a new record in an Airtable table. AIRTABLE_API_KEY
TodoistTool Custom Manages tasks and projects in Todoist. TODOIST_API_KEY
WikipediaTools Custom A suite of tools to search and process Wikipedia articles. No
ComposioTool CrewAI Gateway to integrate with over 200 third-party apps. COMPOSIO_API_KEY

3. Model Context Protocol (MCP)

No MCP servers are used. Wikipedia, formerly an MCP server for deep_research and pestel, now comes from the in-process WikipediaSearchTool and WikipediaArticleTool (see the table above).