Skip to content

Commit 609a959

Browse files
hkong-mitrehkong-mitre
authored
Merge pull request #36 from CVEProject/dev
merge code deployed to PROD from dev to main
2 parents 12562f2 + 05363e2 commit 609a959

File tree

112 files changed

+197711
-2709
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

112 files changed

+197711
-2709
lines changed

‎ChangeLog.md‎

Lines changed: 51 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,12 @@
11
# Change Log
22

3-
## 2.0.0-rc14
4-
- initial version of `cve-core` as a peer project to other `cve-projects`. Can be used as part of a monorepo
3+
### 2.1.0
4+
- wildcard search using "*" and "?"
5+
- AppConfig to manage hierarchical environment variables and all default values for environment variables
6+
- code refactoring: reorganizing code, general cleaning up for migration to github
7+
8+
### 2.0.0-rc14 - deployed: 2025-06-05
9+
- initial version of `cve-core` as a peer project to other `cve-projects`. Can be used as part of a monorepo (e.g., https://github.com/CVEProject/CVE-Search-API/tree/dev)
510
- search using `axios`, NodeJS-native `fetch` and `@opensearch-project/opensearch` libraries
611
- CVE-, CWE-, and CAPAC- IDs
712
- CVE YEAR
@@ -12,7 +17,6 @@
1217
- hyphenated words (e.g., "man-in-the-middle")
1318
- software names (e.g., "Node.JS", ".NET")
1419
- file extension (e.g., "matvar_struct.c")
15-
- repeating non-language characters (e.g., "aaaaa" is ok, but "?????" is replaced by "")
1620
- can run as AWS Lambda Layer
1721
- new adapters
1822
- CVE Services reader
@@ -26,18 +30,47 @@
2630

2731
## Older Milestones from the older `cveUtils`/`cvelist-bulk-download` repositories
2832

29-
Note that the following milestones were in other repositories, which contained a superset of the source code in this npm library. The milestones below are meant only for historic reference, in case a full history of an implementation is needed.
33+
Note that the following milestones were in multiple repositories, and together contained a superset of the source code in this npm library. The milestones below are meant only for historic reference, in case a full history of an implementation is needed.
34+
35+
### 1.3.0 - deployed only on AWS in 2024-12 for initial search capability (tag `2024-12-06`)
36+
- search using `axios` and `@opensearch-project/opensearch` libraries
37+
- general search for tokenized strings in all fields
38+
- CVE-ID
39+
40+
### 2.0.0-rc14
41+
- initial version of `cve-core` as a peer project to other `cve-projects`. Can be used as part of a monorepo (e.g., https://github.com/CVEProject/CVE-Search-API/tree/dev)
42+
- search using `axios`, NodeJS-native `fetch` and `@opensearch-project/opensearch` libraries
43+
- CVE-, CWE-, and CAPAC- IDs
44+
- CVE YEAR
45+
- basic version strings (e.g., "v3.2.5", "v3.2.5-RC1")
46+
- basic IPv4 and IPv6
47+
- URLs
48+
- compound words (e.g., "docker-compose", "microsoft word")
49+
- hyphenated words (e.g., "man-in-the-middle")
50+
- software names (e.g., "Node.JS", ".NET")
51+
- file extension (e.g., "matvar_struct.c")
52+
- repeating non-language characters (e.g., "aaaaa" is ok, but "?????" is replaced by "")
53+
- can run as AWS Lambda Layer
54+
- new adapters
55+
- CVE Services reader
56+
- CVE Search reader
57+
- CVE file reader
58+
- file reader/writer
59+
- console input for interacting with a user in a CLI
60+
- CveResult class with standardized errors and messages (this version is aimed at the search service)
61+
- object (JSON) comparer using `json-difference` library
62+
- JSON replacer that alphabetizes keys when serializing using JSON.stringify()
3063

3164
### 1.2.0 - deployed 2024-07-18 (tag `2024-07-18_v1.2.0`)
3265
- baseline for the `cve-core` npm library
3366
- changes for cisa adp, reference ingest
34-
- axios-retry for network retry
35-
- optimized update.yml to use fetch-depth: 1
36-
- CVES_MAX_ALLOWABLE_CVE_YEAR environment variable set to 2025
37-
- GIT_MAX_FILESIZE_MB environment variable set to 100
67+
- `axios-retry` library for network retry
68+
- optimized `update.yml` to use `fetch-depth: 1`
69+
- `CVES_MAX_ALLOWABLE_CVE_YEAR` environment variable set to 2025
70+
- `GIT_MAX_FILESIZE_MB` environment variable set to 100
3871
- initial refactoring of core classes to separate I/O functions from business logic classes (work in progress)
39-
- minimized 3rd party dependency in IsoDateString class to minimize footprint for AWS Lambda
40-
- import specific lodash functions instead of the full lodash to minimize footprint for AWS Lambda
72+
- minimized 3rd party dependency in IsoDateString class to minimize AWS Lambda footprint
73+
- import specific lodash functions instead of the full lodash to minimize AWS Lambda footprint
4174
- dependabot PRs defaults to develop branch
4275
- cveUtils/GitLab PR 32
4376

@@ -46,31 +79,30 @@ Note that the following milestones were in other repositories, which contained a
4679
- tested but not used on cvelistV5
4780

4881
### 1.1.0 - 2023-09-26 (tag `2023-09-26_v1.1.0`)
49-
- Delta files in /cves (delta.json and deltaLog.json), replacing recent_activities.json
82+
- Official support for delta files in /cves (delta.json and deltaLog.json), replacing recent_activities.json
5083

5184
### 1.0.0 - 2023-05-26 (tag `2023-04-25_v1.0.0`)
5285
- Official version using public domain code in https://github.com/CVEProject/cvelist-bulk-download
5386

54-
5587
### `Sprint-0` - 2023-04-20 (tag `2023-04-20_initial_cveUtils_on_github`)
56-
- initial version selectively copied from internal MITRE gitlab to https://github.com/hkong-mitre/cvelist-bulk-download
57-
- https://github.com/hkong-mitre/cvelist-bulk-download/commit/207b9f2b82908afbd8d9d2270969f6781f9d39e4
58-
- (note date is different): https://gitlab.mitre.org/hkong/cve_utils/-/tags/2023-04-25_to_github_hkong-mitre_cvelist_bulk_download
88+
- initial version selectively copied from internal MITRE gitlab to https://github.com/CVEProject/cvelist-bulk-download
89+
- https://github.com/CVEProject/cvelist-bulk-download/commit/207b9f2b82908afbd8d9d2270969f6781f9d39e4
90+
- note date is slightly different in GitLab tag in cve_utils, but the code is functionly the same: `2023-04-25_to_github_hkong-mitre_cvelist_bulk_download`
5991

6092

6193
### 2023-03-29
62-
- official version used in GitHub actions that updated /cves when cvelistV5 was announced at CNA Summit 2023
63-
- https://gitlab.mitre.org/hkong/cve_utils/-/tags/2023-03-29-cveproject_cvelistV5_dist_(similar)
94+
- official version used in GitHub actions that updated `/cves` when cvelistV5 was announced at CNA Summit 2023
95+
- GitLab tag in cve_utils: `2023-03-29-cveproject_cvelistV5_dist_(similar)`
6496

6597

6698
### 2023-03-10
6799
- code during team code walkthru
68-
- https://gitlab.mitre.org/hkong/cve_utils/-/tags/2023_03_10_code_walkthrough_with_team
100+
- GitLab tag in cve_utils: `2023_03_10_code_walkthrough_with_team`
69101

70102

71103
### 2023-03-06
72104
- first version deployed to cvelistV5 for testing (using `preview_cves` instead of `cves`)
73-
- https://gitlab.mitre.org/hkong/cve_utils/-/tags/2023_03_06_deployed_to_cveproject_cvelistv5
105+
- GitLab tag in cve_utils: `2023_03_06_deployed_to_cveproject_cvelistv5`
74106

75107

76108
## Additional Information

‎config/custom-environment-variables.jsonc‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
{
2+
// this file is used by node-config to map a node-config (AppConfig)
3+
// hierarchy of constants to an environment variable
4+
// Note that much of the environment variables mapped here existed for some time
5+
// without AppConfig, this file bridges the historical uses of those with the new
6+
// as we transition to AppConfig
7+
"appConfig": {
8+
// constants for search capability
9+
"search": {
10+
"providerEndpoint": "OpenSearchDomainEndpoint",
11+
"index": "OpenSearchCveIndex",
12+
// allows local development using containers that do not have SSL certs
13+
"allowUnknownSslCerts": "OpenSearchAllowUnknownSslCerts"
14+
},
15+
// constants for testing node-config
16+
// these values are only used to test node-config in AppConfig.test.int.ts
17+
// DO NOT USE THIS FOR ANYTHING ELSE
18+
"test": {
19+
"appConfigTest": {
20+
"test": "JEST_env_config_test"
21+
}
22+
}
23+
}
24+
}

‎config/default.jsonc‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
{
2+
// Default configuration
3+
// These values are overridable using other *.jsonc (e.g., prod.jsonc)
4+
// as well as using environment variables (e.g., in `.env`)
5+
// - Each configuration is mapped using custom-environment-variables.jsonc to enable environment varialbe overrides.
6+
// - For more information, see cve-core/src/adapters/config/AppConfig.ts
7+
// NOTE for consistency, all values need to be strings for proper type when using .env overrides
8+
"appConfig": {
9+
// constants for search capability
10+
"search": {
11+
// minimum versions for servers that are compatible with current code
12+
"minServer": [
13+
"elasticsearch:7.10.2",
14+
"opensearch:2.10.0"
15+
],
16+
// setting this to FALSE (recommended) requires an SSL cert to access the search server
17+
// The only time this should be allowed to be true is when developing or testing
18+
// using containers that do not have SSL certs
19+
"allowUnknownSslCerts": "FALSE"
20+
}
21+
}
22+
}

‎config/devel.jsonc‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
{
2+
// development configurations
3+
// overrides values specified in default.jsonc, read additional comments there
4+
// and in cve-core/src/adapters/config/AppConfig.ts
5+
// NOTE for consistency, all values need to be strings for proper type when using .env overrides
6+
"appConfig": {
7+
// constants for search capability
8+
"search": {
9+
// minimum versions for servers that are compatible with current code
10+
"minServer": [
11+
"opensearch:2.10.0"
12+
],
13+
// URL to reach search server
14+
"providerEndpoint": "https://admin:admin@localhost:9200",
15+
// index on search server related to searching CVEs
16+
"index": "e2e-cve-test-index-1109",
17+
// setting this to FALSE (recommended) requires an SSL cert to access the search server
18+
// The only time this should be allowed to be true is when developing or testing
19+
// using containers that do not have SSL certs
20+
// DO NOT USE THIS IN ANY PUBLIC OR PRODUCTION ENVIRONMENTS
21+
"allowUnknownSslCerts": "TRUE"
22+
},
23+
// constants for unit, int, e2e testing
24+
"test": {
25+
// constants for testing search capability
26+
"searchTest": {
27+
// many tests for search uses snapshots, which requires CVEs to remain unchanged
28+
// since the live server is updated all the time, a fixture containing fixed CVEs
29+
// is required to keep the test consistent. "fixtures" provides the link
30+
// to the cve-fixtures repository
31+
"fixtures": {
32+
// @todo these constants needs to be in sync in cve-fixtures
33+
// so that testing snapshots are consistent and valid
34+
"name": "fixtures-search-baseline-1086", // release tag
35+
"numCves": "1086" // possible identifier assuming we always add cves to a new release
36+
}
37+
},
38+
// constants for testing node-config
39+
"appConfigTest": {
40+
// these values are only used to test node-config in AppConfig.test.int.ts
41+
// DO NOT USE THIS FOR ANYTHING ELSE
42+
"two": "2",
43+
"five": "5"
44+
}
45+
}
46+
}
47+
}

‎config/prod.jsonc‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
{
2+
// production (example) constants
3+
// overrides values specified in default.jsonc, read additional comments there
4+
// and in cve-core/src/adapters/config/AppConfig.ts
5+
// NOTE for consistency, all values need to be strings for proper type when using .env overrides
6+
"appConfig": {
7+
"search": {
8+
"minServer": [
9+
"elasticsearch:7.10.2",
10+
"opensearch:2.10.0"
11+
],
12+
"providerEndpoint": "", // preference is to specify this in a (secret) environment variable on production platforms
13+
"index": "", // preference is to specify this in a (secret) environment variable on production platforms
14+
"allowUnknownSslCerts": "FALSE"
15+
}
16+
}
17+
}

‎docs/BasicSearchManager.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,23 +1,23 @@
11
# BasicSearchManager
22

33
`BasicSearchManager` provides basic facilities for working with an ElasticSearch/OpenSearch instance. It provides the following:
4-
- `search` providing a standardized way to do a search. This method hides the details of how a search is done depending on the user's search text and other input parameters (e.g., when doing faceted search or when all matches are requested, requiring paging). When using this asynchronous method, the returned `CveResult` will contain results from the search, plus possibly notes and errors that were found during data validation and searching.
5-
- `validateSearchText` is a synchronous method that will return a `CveResult` object potentially containing notes and errors.
4+
- `search()` providing a standardized way to do a search. This method hides the details of the different ways a search is carried out in ElasticSearch/OpenSearch using the user's search text and other input parameters (e.g., when doing faceted search or when all matches are requested, requiring paging). When using this asynchronous method, the returned `CveResult` will contain results from the search, plus possibly notes and errors that were found during data validation and searching.
65

76
In addition, the following associated classes and types are also defined:
8-
- `SearchProviderInfo` --- an object to fully represent a specific index in an ElasticSearch/OpenSearch instance
9-
- `SearchOptions` --- options when searching
7+
- `SearchProviderSpec` --- an object to fully represent a specific index in an ElasticSearch/OpenSearch instance
8+
- `SearchOptions` --- options when searching (e.g., `default_operator`)
109
- `SearchResultData` --- a strongly typed type to facilitate working with search results
1110

12-
For an example of how to use the BasicSearchManager and its associated classes and types, see [BasicSearchManager Examples](#basicsearchmanager-examples).
11+
For examples of how to use the BasicSearchManager see [Simple Search Example](#simple-search-example).
1312

1413
## BasicSearchManager Examples
1514

1615
### Simple Search Example
1716

1817
```typescript
1918
import { CveResult } from "cve-core/CveResult.js"
20-
import { SearchResultData, BasicSearchManager } from "cve-core/BasicSearchManager.js";
19+
import { BasicSearchManager } from "cve-core/BasicSearchManager.js";
20+
import { SearchResultData } from "cve-core/SearchResultData.js";
2121
const simpleSearch = async () => {
2222
const searchManager = new BasicSearchManager({
2323
index: "cve-index-local",

‎index.ts‎

Lines changed: 39 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -1,52 +1,61 @@
11
/**
2-
* This is intended to be the main export file for cve-core.
2+
* This is the main export file for cve-core when used as library
3+
* Files that makes up this library should not use this, however,
4+
* and should use relative paths
35
*/
46

57
// adapters
6-
export * from "./src/adapters/fs/CveFsReader.js"
7-
export * from "./src/adapters/fs/FsReader.js"
8-
export * from "./src/adapters/fs/FsWriter.js"
9-
export * from './src/adapters/search/SearchAdapter.js'
10-
export * from './src/adapters/search/SearchReader.js'
8+
export * from "./src/adapters/config/AppConfig.js";
9+
export * from './src/adapters/console/ConsoleInputReader.js';
10+
export * from './src/adapters/cveservice/CveService.js';
11+
export * from './src/adapters/cveservice/CveServiceBaseUrl.js';
12+
export * from './src/adapters/cveservice/CveServiceCreds.js';
13+
export * from './src/adapters/cveservice/cve/CveServiceCveReader.js';
14+
export * from './src/adapters/cveservice/healthCheck/CveServiceHealthReader.js';
15+
export * from "./src/adapters/fs/CveFsReader.js";
16+
export * from "./src/adapters/fs/DirectoryWalker.js";
17+
export * from "./src/adapters/fs/FsReader.js";
18+
export * from "./src/adapters/fs/FsWriter.js";
19+
export * from './src/adapters/search/SearchAdapter.js';
20+
export * from './src/adapters/search/SearchReader.js';
21+
1122

1223
// commands
13-
export * from "./src/commands/DateCommand.js"
14-
export * from "./src/commands/GenericCommand.js"
15-
export * from "./src/commands/MainCommands.js"
24+
export * from "./src/commands/DateCommand.js";
25+
export * from "./src/commands/GenericCommand.js";
26+
export * from "./src/commands/MainCommands.js";
1627

1728
// common
18-
export * from "./src/common/IsoDate/IsoDateString.js"
19-
export * from "./src/common/Json/Json.js"
20-
export * from "./src/common/comparer/ObjectComparer.js"
29+
export * from "./src/common/IsoDate/IsoDateString.js";
30+
export * from "./src/common/Json/Json.js";
31+
export * from "./src/common/comparer/ObjectComparer.js";
2132

2233
// core
23-
export * from "./src/core/result/CveResult.js"
24-
export * from "./src/core/CveId.js";
34+
export * from "./src/cveId/CveId.js";
2535
export * from "./src/core/Activity.js";
2636
export * from "./src/core/ActivityLog.js";
27-
export * from "./src/core/CveComparer.js";
28-
export * from "./src/core/CveCore.js";
29-
export * from "./src/core/CveCorePlus.js";
30-
export * from "./src/core/CveDate.js";
31-
export * from "./src/core/CveId.js";
32-
export * from "./src/core/CveListDir.js";
33-
export * from "./src/core/CveRecord.js";
37+
export * from "./src/common/comparer/CveComparer.js";
38+
export * from "./src/cve/CveCore.js";
39+
export * from "./src/cve/CveCorePlus.js";
40+
export * from "./src/date/CveDate.js";
41+
export * from "./src/deprecated/CveListDir.js";
42+
export * from "./src/cve/CveRecord.js";
3443
export * from "./src/core/Delta.js";
3544
export * from "./src/core/DeltaFs.js";
3645
export * from "./src/core/DeltaLog.js";
37-
export * from "./src/core/fsUtils.js";
46+
export * from "./src/deprecated/fsUtils.js";
3847
export * from "./src/core/git.js";
48+
export * from "./src/date/CveDate.js";
3949

40-
//search
41-
export * from './src/core/search/BasicSearchManager.js'
50+
// cve result
51+
export * from "./src/result/CveResult.js";
4252

43-
// generated
44-
export * from "./src/generated/quicktools/CveRecordV5.js";
53+
// search
54+
export * from './src/search/BasicSearchManager.js';
55+
export * from './src/search/SearchRequest.js';
4556

46-
// net
47-
export * from "./src/net/ApiBaseService.js";
48-
export * from "./src/net/CveService.js";
49-
export * from "./src/net/CveUpdater.js";
57+
// generated
58+
export * from "./src/cve/record/generated/CveRecordV5.js";
5059

5160
// package info
5261
import * as packageJSON from './package.json';

‎jest.config.js‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
export default {
22
testEnvironment: 'node',
3-
preset: 'ts-jest/presets/default-esm',
3+
preset: 'ts-jest/presets/js-with-ts-esm',
44
globals: {
55
'ts-jest': {
66
useESM: true,
@@ -22,4 +22,7 @@ export default {
2222
'!src/**/*.d.ts',
2323
'!src/**/*.d.mts',
2424
],
25+
setupFilesAfterEnv: [
26+
'<rootDir>/src/jest.setup.ts'
27+
]
2528
};

0 commit comments

Comments
 (0)