Analyst

An agent that answers questions about cybersecurity conference research using an agent curated knowledge base. UI is via Jupyter notebook or command prompt.SQL queries can also be used to search for researchers, talks, or tools.

Answers to free-form questions are given from three dozen pre-processed topic summaries compiled by a knowledge agent. There are also summaries for each conference and for authors with more than one presentation in the data.

Colab instance here: https://colab.research.google.com/github/opendr-io/analyst/blob/main/the-analyst.ipynb#scrollTo=d2863aa6

User Interface:

Question - answers questions using the pre-processed topic summaries from the conference list above. This requires an OpenAI key. Question types:

• Free form questions like "give me a complete report on vulnerability research" are answered from the 36 topic summaries compiled across every conference in the database.
• Conference questions like "tell me all about defcon 34" or another con in the data (see list below)

Query - runs keyword searches for talks, tools, researchers, etc if you know what you're after - deterministic annotation/database SQL queries and does not call an LLM, no key required.

Topics - lists the topic names; each topic has a topic summary created from the entire record set.

Conferences in the data:

• All years: CAMLIS, PROMPT|GTFO, Unprompted.
• 2025 conferences: DEF CON, Blackhat, BSides LV.
• 2026 conferences: DEF CON, Blackhat, BSides LV.

Quickstart

1. Set up Python dependencies

Use a virtual environment if possible.

python -m venv analyst
python -m pip install -r requirements.txt

2. Configure API keys

Create or update .env in the repo root.

OPENAI_API_KEY=...
ANTHROPIC_API_KEY=...

The active LLM provider and model names are configured in config/llm.ini.

3. Verify the database

The default database is:

knowledge/knowledge.sqlite3

Run a quick search:

python -m knowledge_indexing.knowledge_index --query "threat modeling" --limit 5

Audit imported source counts against local source artifacts:

python tools\source_count_auditor.py --strict

Common Workflows

Ask questions

the_analyst is the primary interface. See docs/analyst.md for its tool routing, evidence rules, author behavior, configuration, and failure modes.

Interactive mode:

python the_analyst.py

Ask one question from the command line:

python the_analyst.py "question: How is threat modeling being done using AI?"

Use question: when the answer should come from generated summary files. analyst routes these requests to up to three relevant summaries by default and does not fill gaps with general knowledge. Use query: when you want deterministic keyword search over record annotations.

python the_analyst.py "query: cryptocurrency wallet fraud"

The QA agent should cite which summary files or annotation query results it used.

Analyst Jupyter notebook UI:

the-analyst.ipynb

The analyst notebook exposes the constrained question: path, deterministic query: path, and status check.

Output Locations

Primary working data:

knowledge/knowledge.sqlite3

Generated reports and summaries:

summaries/
knowledge/open-topic-report.md
knowledge/exports/

Logs and status:

knowledge/import.log
knowledge/classify-run.log

Prompt files:

knowledge_agenting/prompts/

Topic summary design notes:

docs/topic-summary-design.md

Safety Checks

Before large write operations, make a SQLite backup:

Copy-Item knowledge\knowledge.sqlite3 knowledge-backup-$(Get-Date -Format yyyyMMdd-HHmmss).sqlite3

Run read-only summary dry-runs before expensive LLM calls:

python -m knowledge_agenting.topic_summarizer --group-by topic --topic "Threat modeling" --dry-run

Run focused tests after code changes:

python -m pytest -q tests\test_topic_summarizer.py
python -m pytest -q tests\test_knowledge_qa.py

Useful Review Commands

Count or inspect annotations:

python -m knowledge_indexing.knowledge_index --latest-annotations --limit 20
python -m knowledge_indexing.knowledge_index --list-annotations --annotation-query "cryptocurrency" --limit 20

Inspect classifications:

python -m knowledge_indexing.knowledge_index --list-classifications --limit 20
python -m knowledge_indexing.knowledge_index --low-confidence-classifications --limit 50

List records for a topic:

python -m knowledge_indexing.knowledge_index --list-topic-records "Threat modeling" --limit 50

Generate long-form summaries

Use the knowledge_agenting.topic_summarizer module for the current spec-compliant summary generator. It can summarize by topic, source, or author.

Dry run one topic without calling the LLM or writing artifacts:

python -m knowledge_agenting.topic_summarizer --group-by topic --topic "Threat modeling" --dry-run

Write preflight prompt/audit/manifest artifacts during a dry-run:

python -m knowledge_agenting.topic_summarizer --group-by topic --topic "Threat modeling" --dry-run --write-preflight-artifacts

Generate one topic summary:

python -m knowledge_agenting.topic_summarizer --group-by topic --topic "Threat modeling"

Generate all topic summaries with parallel workers:

python -m knowledge_agenting.topic_summarizer --group-by topic --all --parallel 4

List existing topic summaries:

python -m knowledge_agenting.topic_summarizer --group-by topic --list-summaries

List topics that currently have classified records and can be summarized:

python -m knowledge_agenting.topic_summarizer --group-by topic --list-topics

Generate source summaries:

python -m knowledge_agenting.topic_summarizer --group-by source --all --parallel 4

Generate author summaries for authors with at least two records:

python -m knowledge_agenting.topic_summarizer --group-by author --all --parallel 4

Generate only missing author summaries and leave existing author summaries untouched:

python -m knowledge_agenting.topic_summarizer --group-by author --all --skip-existing --parallel 4

List missing author summaries without generating or writing anything:

python -m knowledge_agenting.topic_summarizer --group-by author --all --list-missing

Author summaries use the summarize_author model from config/llm.ini. Use --min-records 1 explicitly to include single-record authors. Coauthored records are included in each individual author's summary; comma, semicolon, ampersand, and and separators are expanded during author grouping.

Individual authors are normalized in SQLite:

authors             unique author names
author_records      author-to-record links

records.author retains the original imported author string for provenance.

Summary output files are written to:

summaries/topics/*.md
summaries/sources/*.md
summaries/authors/*.md

Operational artifacts are kept separate:

summaries/artifacts/topics/*
summaries/artifacts/sources/*
summaries/artifacts/authors/*

Artifacts include prompt inputs, audit JSON, manifest JSON, and archived prior outputs. Topic summaries are stored as Markdown files; SQLite does not store the summary bodies.