Skip to content

Commit 959b3ce

Browse files
authored
feat: jh search command (code, symbols, docs, packages) with --json (#62)
* feat: `jh search` command (code, symbols, docs, packages) with --json Calls JuliaHub's versioned search API (POST /search/v1/{code,sym,docs}) and prints a table, or with --json only the {results, truncated} document on stdout so agents and scripts can consume it. Filters: --package (name or UUID, resolved via /packages/info), --registry, --path, --ignore-case, --type, --usage, --threshold, --strict-phrase, --limit. Errors are the server's {code, message} on stderr with exit 1; invalid_registry prints the registries the instance knows. Installs that predate /search/v1 answer 404 there; the command falls back to the legacy /search/* route and decodes its {success, data} envelope. --server accepts a full URL (http://localhost:PORT) for local development; an unauthenticated request is sent when no token is stored for an http URL. jh search packages is jh package search under the new group; both gain --json, and the REST -> GraphQL fallback now warns on stderr as CLAUDE.md already claimed. Registry-name resolution is shared (resolveRegistries). Refs JuliaComputing/JuliaHub#24098 * fix: resolve --package names via GraphQL when /packages/info is unavailable Mirrors getPackageInfo's REST -> GraphQL fallback so jh search works on installs without the packages REST endpoint. * fix(search): never send the token in cleartext, and fall back to legacy only when v1 is absent - The stored JuliaHub token is sent over https, or over plain http only to a loopback host. Any other http:// --server gets an unauthenticated request and a note on stderr, instead of the juliahub.com ID token in cleartext. - A 404/405 from the v1 route whose body is a structured {code, message} error is reported as that error. Only an unstructured 404/405 (a web server without the route) triggers the legacy retry, so a structured error is neither masked by a second search nor reported as search_unavailable. - Package search fetches over REST and falls back to GraphQL only when the fetch fails; the results are emitted once, after fetching. A stdout write error (closed pipe in --json mode) no longer re-runs the search and appends a second document. Addresses the review on #62.
1 parent 509c769 commit 959b3ce

11 files changed

Lines changed: 1881 additions & 143 deletions

File tree

‎CLAUDE.md‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ The application follows a command-line interface pattern using the Cobra library
2222
- **landing.go**: Landing page management (show, update, remove) via the homepage layout API. The layout is a JSON widget array; the landing page is the `greeter` widget's `metadata.content` markdown. `update` replaces that content (adding a greeter as the first, full-width row and shifting the others down when none exists), `remove` drops the greeter widget; both POST the complete layout back, preserving every field the CLI does not model. There is no separate DELETE endpoint any more
2323
- **vuln.go**: Vulnerability scanning for Julia packages via REST API
2424
- **scan.go**: Trivy manifest scanning — discover and upload a Julia manifest (+ optional project file) and fetch results via the StaticAnalysis service. Manifest discovery recognizes every name Julia/Pkg accepts (`Manifest.toml`, `JuliaManifest.toml`, and version-specific `Manifest-v1.11.toml` / `JuliaManifest-v1.11.toml`); when a directory holds more than one, the user is prompted to select (defaults to the highest-precedence candidate). Project file is `Project.toml` or `JuliaProject.toml`
25+
- **search.go**: `jh search code|symbols|docs` against the search service (`POST /search/v1/{code,sym,docs}`, legacy `POST /search/{code,sym,docs}` fallback), with request builders, response decoders, table renderers and the `--json` writer as pure functions; the cobra commands live in `main.go`
2526
- **git.go**: Git integration (clone, push, fetch, pull) with JuliaHub authentication
2627
- **julia.go**: Julia installation and management
2728
- **run.go**: Julia execution with JuliaHub configuration
@@ -64,6 +65,7 @@ The application follows a command-line interface pattern using the Cobra library
6465
- `jh admin landing-page`: Landing page management (show/update/remove the home page's Welcome/greeter markdown card via the homepage layout API)
6566
- `jh vuln`: Vulnerability scanning for Julia packages (REST API; defaults to latest stable version via `GET /docs/<registry>/<package>/versions.json`; supports `--version` for a specific version, `--registry` to specify the registry for version lookup (default: General), `--advisory` to filter to a specific advisory ID, `--all` to show all advisories regardless of affected status, and `--verbose` for additional details)
6667
- `jh scan`: Manifest vulnerability scanning via the StaticAnalysis service (REST API under `/api/v0/static_analysis/`); uploads a `Manifest.toml` (and sibling `Project.toml` unless `--no-project`) and POSTs to `/manifest/scan` to obtain a `run_uuid`; subcommands `status` and `results` hit `/manifest/scan/status/{run_uuid}` and `/results/manifest/{run_uuid}`; **polls by default** and prints results when the scan completes — Ctrl+C detaches and the `run_uuid` (printed up front) can be passed to `jh scan status` / `jh scan results` later. Pass `--no-wait` to submit-and-exit. `--csv` requests `text/csv` (content-negotiated via `Accept` header)
68+
- `jh search`: Search across indexed packages — `code <pattern>` (RE2 over source files; `--path`, `--ignore-case`), `symbols <name>` (`--type function|type|macro|module`, `--usage define|use`, validated client-side), `docs <query>` (`--threshold`, `--strict-phrase`), all with repeatable `--package` (name or UUID) / `--registry`, `--limit` and `--json`; `packages [term]` is `jh package search` under the search group (shares `runPackageSearch`)
6769
- `jh clone`: Git clone with JuliaHub authentication and project name resolution
6870
- `jh push/fetch/pull`: Git operations with JuliaHub authentication
6971
- `jh git-credential`: Git credential helper for seamless authentication
@@ -252,6 +254,19 @@ go run . scan status <run-uuid>
252254
go run . scan results <run-uuid> --csv
253255
```
254256

257+
### Test search operations
258+
```bash
259+
go run . search code "function foo" --limit 5
260+
go run . search code "@kwdef" --package DataFrames --path "^src/" --ignore-case
261+
go run . search symbols DataFrame --type type --usage define
262+
go run . search docs "join two tables" --limit 5 --threshold 0.5
263+
go run . search packages dataframes --registries General
264+
go run . search code "innerjoin" --json | jq .truncated
265+
266+
# Against a locally running search service (no login needed; --package must be a UUID)
267+
go run . search code "foo" --server http://localhost:4446
268+
```
269+
255270
### Test Git operations
256271
```bash
257272
go run . clone john/my-project # Clone from another user
@@ -330,6 +345,7 @@ The application uses OAuth2 device flow:
330345
- **Registry credentials**: `GET /api/v1/sysconfig/credentials` to fetch (returns `CredentialsInfo` object directly); `POST` to add tokens/apps; `PUT` to update tokens/apps or replace all SSH credentials; `DELETE` with `{tokens:[...], githubApps:[...]}` to delete
331346
- **Package search/info primary**: `/packages/info` endpoint with `name`, `registries`, `tags`, `licenses`, `limit`, `offset` query params; returns `{packages: [...], meta: {total: N}}`
332347
- **Package dependencies**: `/docs/{registry}/{package}/stable/pkg.json` for dependency information
348+
- **Search**: `POST /search/v1/code`, `/search/v1/sym`, `/search/v1/docs` with a JSON body (`pattern`, `package` [UUIDs], `registry` [names], `maxresults`, plus per-kind `pathfilter`/`ignorecase`, `types`/`usages`, `threshold`/`strictphrase`); 200 → `{"results": [...], "truncated": bool}`, any failure → non-2xx `{"code", "message", "details"?}` with stable codes (`invalid_request`, `request_too_large`, `invalid_pattern`, `invalid_pathfilter`, `invalid_types`, `invalid_usages`, `invalid_registry`, `search_failed`, `metadata_unavailable`). Older installs have only `POST /search/{code,sym,docs}`, which answer 200 with `{"success": bool, "data": <results or error string>}`; the CLI retries there when v1 answers 404/405
333349
- **Authentication**: Bearer token with ID token
334350
- **Upload workflow**: 3-step process (request presigned URL, upload to URL, close upload)
335351
- **Credential write pattern**: Targeted mutations via `POST`/`PUT`/`DELETE`; SSH operations require read-modify-write (fetch + full-replacement `PUT`) since `sshcreds` in PUT is a full replacement
@@ -508,7 +524,13 @@ jh run setup
508524
- Landing page commands (`jh admin landing-page`) use REST API: GET `/app/homepage` (show), POST `/app/config/homepage` (update), DELETE `/app/config/homepage` (remove); require appropriate permissions
509525
- Landing page `update` command accepts content inline as an argument, from a file via `--file`, or piped via stdin (priority: `--file` > arg > stdin)
510526
- Landing page response uses custom JSON unmarshaling (`homepageResponse`) to handle `message` being either an object or a string
511-
- Package search (`jh package search`) and info (`jh package info`) both try REST API (`/packages/info`) first, then fall back to GraphQL (`FilteredPackages` / `FilteredPackagesCount` via `/v1/graphql`) on failure; a warning is printed to stderr when the fallback is used
527+
- Package search (`jh package search`) and info (`jh package info`) both try REST API (`/packages/info`) first, then fall back to GraphQL (`FilteredPackages` / `FilteredPackagesCount` via `/v1/graphql`) on failure; `searchPackages` prints `warning: REST package search failed (...); falling back to GraphQL` to stderr when it falls back (`getPackageInfo` falls back silently)
528+
- `--registries` resolution for `jh package search` / `jh package info` / `jh search packages` goes through `resolveRegistries(server, csv)` in `registries.go` (empty = all registries; unknown name → `Registry not found: '<name>'`)
529+
- `jh package search` / `jh search packages --json` sets `PackageSearchParams.JSON`; both the REST and GraphQL paths then call `writePackagesJSON`, which prints `{"results": [...], "total": N}` (snake_case keys from `packageInfo`'s json tags, `tags` always an array) 2-space indented with a trailing newline — nothing else on stdout
530+
- **JSON output convention** (`jh search *`): `--json` prints only the document on stdout, 2-space indented, trailing newline — `{"results": <server array, untouched>, "truncated": bool}` for code/symbols/docs; warnings, the human-mode `note: results truncated (search stopped early)` line, and errors always go to stderr; exit code is 0/1 only
531+
- `jh search` HTTP flow: `postSearch(baseURL, kind, body)` → `postSearchWithToken(client, baseURL, kind, token, body)` → `doSearchRequest(client, baseURL, path, token, body)`; v1 first, legacy route on 404/405, `search_unavailable` when both are missing. Non-2xx bodies become `*searchError{Status, Code, Message, Details}` (`Error()` is `<code>: <message>`, or `HTTP <status>: <body>` when undecodable; legacy `success:false` → code `legacy_error`); `printSearchError` adds a `known registries: ...` line from `details.known` on `invalid_registry`
532+
- `--server` for `jh search code|symbols|docs` is resolved by `searchBaseURL`: an explicit `http://`/`https://` URL is used verbatim (trailing slash trimmed) so the commands can target a local search service (`--server http://localhost:4446`); anything else is `https://` + `normalizeServer`. Against an `http://` base a missing stored token is tolerated (request sent without `Authorization`, note on stderr) and `--package` must be a UUID since name lookup needs `/packages/info`; `https://` still requires `ensureValidToken()`
533+
- `--package` names are resolved with `fetchRESTPackages(server, name, 100, 0, nil)` + exact case-insensitive match; the same package appears once per registry with the same UUID, so UUIDs are deduplicated, and distinct packages sharing a name all contribute; no match → `package not found: <name>`. `--registry` names are passed through verbatim for the server to validate
512534
- REST API passes `--registries` as comma-separated registry names to the `registries` query param; GraphQL fallback passes registry IDs to the `registries` variable
513535
- `fetchRegistries` in `registries.go` is used by `listRegistries`, `packageSearchCmd`, `packageInfoCmd`, and `packageDependencyCmd` to resolve registry names to IDs (for GraphQL) and names (for REST)
514536
- Both REST and GraphQL package search/info paths produce identical output columns (Registry and Owner); GraphQL resolves registry names from the `registryIDs`/`registryNames` already in `PackageSearchParams` — no extra API call needed

‎README.md‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ A command-line interface for interacting with JuliaHub, a platform for Julia com
1414
- **User Management**: Display user information, list users and groups
1515
- **Vulnerability Scanning**: Scan Julia packages for known security vulnerabilities
1616
- **Manifest Scanning**: Upload a `Manifest.toml` for a server-side Trivy vulnerability scan of its pinned dependencies
17+
- **Search**: Search package source code (regex), symbol definitions and uses, and rendered documentation across the indexed packages, with `--json` output for scripts and agents
1718
- **Administrative Commands**: Manage users, groups, tokens, credentials, and system resources (requires admin permissions)
1819

1920
## Installation
@@ -162,6 +163,7 @@ go build -o jh .
162163
- `--registries <names>` - Filter by registry names (comma-separated, e.g. `General,MyRegistry`)
163164
- `--limit <n>` - Maximum results to return (default: 10)
164165
- `--offset <n>` - Number of results to skip
166+
- `--json` - Print only `{"results": [...], "total": N}` on stdout (also available as `jh search packages`)
165167
- `jh package info <package-name>` - Get detailed information about a specific package (exact name match, case-insensitive)
166168
- `jh package info --registries General` - Search in specific registries only
167169
- `jh package dependency <package-name>` - List package dependencies
@@ -288,6 +290,29 @@ The landing page is the markdown "Welcome" (greeter) card of the home page layou
288290
- `--csv` - CSV output instead of JSON
289291
- `--output <file>` / `-o` - Write to a file instead of stdout
290292

293+
### Search (`jh search`)
294+
295+
Search the Julia packages indexed on JuliaHub. Every subcommand takes `--json`, which prints only a JSON document on stdout (notes and errors go to stderr), so the output can be consumed directly by scripts and by tool-calling agents.
296+
297+
- `jh search code <pattern>` - Regular-expression (RE2) search over package source files; one result per matching line with package, registry, `file:line` and the line text
298+
- `--package <name|uuid>` - Restrict to a package (repeatable; names are resolved to UUIDs via the package catalogue)
299+
- `--registry <name>` - Restrict to a registry (repeatable)
300+
- `--path <regex>` - Only search files whose path matches
301+
- `--ignore-case` - Case-insensitive match
302+
- `--limit <n>` - Maximum number of results (default: the server's)
303+
- `--json` - `{"results": [...], "truncated": bool}`
304+
- `jh search symbols <name>` - Find where a function, type, macro or module is defined and used
305+
- `--type <kind>` - `function`, `type`, `macro` or `module` (repeatable)
306+
- `--usage <kind>` - `define` or `use` (repeatable)
307+
- `--package`, `--registry`, `--limit`, `--json` as for `code`
308+
- `jh search docs <query>` - Natural-language search over rendered package documentation; results are packages ranked by score with their matching sections
309+
- `--threshold <x>` - Drop results scoring below `x`
310+
- `--strict-phrase` - Require the query as a phrase rather than separate terms
311+
- `--package`, `--registry`, `--limit`, `--json` as for `code`
312+
- `jh search packages [search-term]` - The package catalogue search (same as `jh package search`), with `--registries`, `--limit`, `--offset`, `--verbose` and `--json` (`{"results": [...], "total": N}`)
313+
- `--server` / `-s` accepts a JuliaHub host as elsewhere, or a full URL such as `--server http://localhost:4446` to point `code`/`symbols`/`docs` at a locally running search service (no stored login required; the request is sent unauthenticated with a note on stderr, and `--package` must then be given as a UUID). The stored token is only ever sent over `https://`, or over `http://` to a loopback host; a plain-http URL on any other host gets an unauthenticated request and a note
314+
- Older JuliaHub installs expose only the legacy search routes; the CLI detects this and falls back to them automatically. In human-readable mode a `note: results truncated (search stopped early)` line on stderr means the server stopped before exhausting the index
315+
291316
### Update (`jh update`)
292317

293318
- `jh update` - Check for updates and automatically install the latest version
@@ -531,6 +556,35 @@ jh scan results <run-uuid>
531556
jh scan results <run-uuid> --csv --output results.csv
532557
```
533558

559+
### Search
560+
561+
```bash
562+
# Regex search over package source; restrict to two packages and a registry
563+
jh search code "@kwdef" --package DataFrames --package CSV --registry General
564+
565+
# Only files under src/, case-insensitive, at most 20 hits
566+
jh search code "todo" --path "^src/" --ignore-case --limit 20
567+
568+
# Where is the DataFrame type defined?
569+
jh search symbols DataFrame --type type --usage define
570+
571+
# Every use of the @time macro
572+
jh search symbols @time --type macro --usage use --limit 50
573+
574+
# Documentation search, five best packages
575+
jh search docs "read a CSV file into a DataFrame" --limit 5
576+
577+
# Package catalogue search under the search group
578+
jh search packages dataframes --registries General
579+
580+
# Machine-readable output for scripts and agents: only JSON on stdout
581+
jh search code "function innerjoin" --json | jq '.results[] | .packagename + " " + .file'
582+
jh search packages plots --json | jq '.results[0].uuid'
583+
584+
# Point the code/symbols/docs commands at a local search service
585+
jh search code "foo" --server http://localhost:4446 --package a93c6f00-e57d-5684-b7b6-d8193f3e46c0
586+
```
587+
534588
### Git Workflow
535589

536590
```bash

‎e2e/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ One file per command category, plus shared infrastructure:
2727
| [registry_test.go](registry_test.go) | `registry list` → `config`/`registrator`/`permission list` on a real registry |
2828
| [vuln_test.go](vuln_test.go) | `vuln <package>` |
2929
| [scan_test.go](scan_test.go) | `scan <manifest>` (--no-wait) → `scan status`/`results <uuid>` + input validation |
30+
| [search_test.go](search_test.go) | `search packages/symbols/code … --json` (parsed JSON shape) + `search code "["` invalid-pattern error |
3031
| [job_test.go](job_test.go) | `job list` |
3132
| [admin_test.go](admin_test.go) | `admin user/token/group/credential list`, `landing-page show` |
3233
| [verbose_test.go](verbose_test.go) | `--verbose` / flag variants of the read commands (`registry list`, `admin … list`, `package search`, `vuln --all/--verbose`) |

‎e2e/cli_test.go‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ func TestSubcommandHelp(t *testing.T) {
2525
{[]string{"admin", "--help"}, []string{"user", "token", "group", "credential", "landing-page"}},
2626
{[]string{"admin", "credential", "--help"}, []string{"list", "add", "update", "delete"}},
2727
{[]string{"scan", "--help"}, []string{"status", "results"}},
28+
{[]string{"search", "--help"}, []string{"code", "symbols", "docs", "packages"}},
2829
}
2930
for _, tc := range cases {
3031
t.Run(strings.Join(tc.args, " "), func(t *testing.T) {
@@ -46,6 +47,9 @@ func TestArgValidation(t *testing.T) {
4647
{"registry", "config"},
4748
{"scan", "status"},
4849
{"scan", "results"},
50+
{"search", "code"},
51+
{"search", "symbols"},
52+
{"search", "docs"},
4953
}
5054
for _, args := range cases {
5155
t.Run(strings.Join(args, " "), func(t *testing.T) {

0 commit comments

Comments
 (0)