A personal intelligence pipeline that automatically monitors security news, AI research, BSV/Bastion developments and custom topics — summarising and delivering structured briefings directly to Obsidian twice daily.
| Area | Detail |
|---|---|
| Python | Modular design, file I/O, scheduling, API integration, RSS parsing |
| Security Engineering | Domain whitelisting, rate limiting, secrets management, prompt injection mitigation |
| Threat Intelligence | Multi-source ingestion, keyword filtering, CVE/vulnerability feed monitoring |
| AI Integration | Claude API (Anthropic) for automated summarisation with hardened system prompts |
| Automation | Launchd scheduling, deduplication logic, Obsidian vault integration |
| Secure Coding | No hardcoded secrets, .gitignore hygiene, input validation, outbound-only design |
Staying on top of the threat landscape, AI developments and niche research areas is a full time job. This tool automates that process — fetching from trusted sources every 12 hours, filtering by relevance, summarising with Claude AI, and dropping a structured report into your Obsidian vault inbox before you start your day.
- Automated scheduling — runs at 07:00 and 19:00 daily
- Multi-source ingestion — security news, AI research papers, BSV/blockchain feeds
- Keyword filtering — only surfaces content relevant to your defined interests
- Claude API summarisation — each item summarised with: what happened, why it matters, action items
- Prompt injection mitigation — system prompt instructs the model to ignore instructions embedded in article content; not a guarantee, but reduces risk from malicious RSS entries
- Domain whitelist — only fetches from approved, trusted sources
- Rate limiting — respectful scraping, no hammering of external sites
- Obsidian integration — reports drop directly into your vault inbox as structured Markdown
- Local backup — all reports saved locally in
/reports
Each report is categorised and structured for fast consumption:
Each item includes a one sentence summary, why it matters, and action items where relevant.
- API key stored in
.env— never committed to version control - Reports excluded from GitHub via
.gitignore - Domain whitelist prevents fetching from untrusted sources
- Rate limiting between requests prevents abusive scraping
- Claude system prompt instructs the model to ignore instructions in article content — a soft mitigation, not a robust defence against prompt injection
- Outbound only — nothing external can reach your Obsidian vault
Requirements: Python 3.x, Anthropic API key
# Clone the repo
git clone git@github.com:LeightonSec/intel-pipeline.git
cd intel-pipeline
# Create virtual environment
python3 -m venv venv
source venv/bin/activate
# Install dependencies
pip install -r requirements.txt
# Add your API key
echo "ANTHROPIC_API_KEY=your-key-here" > .env
# Run once manually
python3 pipeline.py
# Run on schedule (07:00 and 19:00 daily)
python3 scheduler.pyEdit pipeline.py to customise:
USER_KEYWORDS— add your own topics and interestsOBSIDIAN_PATH— point to your vault inboxRSS_FEEDSinfetcher.py— add or remove sources- Schedule times in
scheduler.py
intel-pipeline/ ├── pipeline.py # Main orchestrator ├── fetcher.py # RSS ingestion, whitelist, rate limiting ├── summariser.py # Claude API summarisation ├── deduplicator.py # Seen URL tracking, prevents duplicate reports ├── scheduler.py # Twice daily automation ├── config/ │ └── sources.yaml # Feed source configuration ├── requirements.txt ├── reports/ # Local report storage (gitignored) ├── logs/ # Runtime logs and seen URL cache (gitignored) └── .env # API key (never committed)
- Web dashboard to view reports in browser
- Slack/email alert for critical security items
- CVE feed integration — CISA alerts, CISA current activity, Exploit-DB
- GitHub trending repos monitoring
- Custom source addition via config file
Leighton Wilson — Security Researcher | LeightonSec LeightonSec GitHub
Built as part of a hands-on cybersecurity portfolio. Part of the LeightonSec security toolkit.