Design proposal: kpt live -- Inventory to ResourceGroup #2576

yuwenma · 2021-11-06T00:50:03Z

This PR is a design proposal on enabling kpt live apply to deploy from STDIN and migrate inventory from Kptfile to a standalone ResourceGroup object/file.

docs/design-docs/01-kpt-live-stdin.md

mikebz

Looks very good. Just a couple of minor questions.

docs/design-docs/01-kpt-live-stdin.md

droot

Thanks for submitting the design here in the repo. I have some minor comments.

docs/design-docs/01-kpt-live-stdin.md

docs/design-docs/01-live_Inventory-to-ResourceGroup.md

mikebz · 2021-11-22T22:25:08Z

@yuwenma seemed like a couple of comments were resolved without answers. If there is a blanket strategy to resolve it - let me know. I think it might be good for people to know why those alternatives were provided or not considered.

mikebz · 2021-11-30T17:07:35Z

🚥 ping....

droot

@yuwenma Proposal is looking in good shape. Two open questions:

suggestion on using short name for the commandline flag. --rg-file is what I have suggested.
I think something we overlooked about the interplay of inventory information with functions such as namespace and ensure-name-substring. High level question to answer is: should kpt fn render or kpt fn eval kpt fn source should ignore the inventory resource or leave it to the functions to ignore it. If we want to maintain the current behavior, then kpt fn should ignore the inventory resource when invoking functions.

docs/design-docs/01-live_Inventory-to-ResourceGroup.md

yuwenma · 2021-12-01T00:24:46Z

@droot Updated to '--rg-file`

phanimarupaka · 2021-12-01T20:33:25Z

@yuwenma Thanks for the doc. Approved it.

@droot We can merge it once you approve.

droot · 2021-12-01T21:42:34Z

@yuwenma I had a discussion with @phanimarupaka today about how to treat inventory resource wrt to kpt fn and kpt pkg commands. Pl. take a look at the summary of the discussion in the last comment. Can you pl. ack the proposal in the comment (below) and would be even better if we can capture this summary in the doc itself somewhere because it impacts the implementation.

#2576 (comment)

phanimarupaka · 2021-12-02T20:22:13Z

@droot @yuwenma I added the commit about the rg file interaction with fn commands.

phanimarupaka · 2021-12-02T21:01:16Z

Merging this PR. Thanks for your contribution @yuwenma. @rquitales Please feel free to start implementing this and let us know if you have any questions.

cc @mikebz

* design document template (#2563) * updated maintainers * update to the front page * installation paragraph * design doc template * additional process detail * slight updates * update kyaml and cli-utils to latest (#2523) * Do not error while adding merge comment (#2564) * Remove check if repo is checked into git for update (#2569) * Remove check if repo is checked into git for update * Update tests * Update should accept empty path with version (#2572) * Update roadmap for Q4 (#2571) * Update roadmap for Q4 * Suggested changes * Update roadmap (#2574) * Update roadmap * Suggested changes * Update with suggested format * Organize better * update installation doc and Homebrew to use 1.0.0-beta.8 (#2578) * Update kpt book with depends-on annotation (#2565) * Update kpt book with depends-on annotation * Addressed comments * Add snippet for resource dependencies in chapter overview * fix: use KptFileKind variable instead of KptFileName for GVR (#2580) When specifying the Kptfile resource kind, some code were using the KptFileName variable instead of KptFileKind. Both variables currently contain the same value: "Kptfile", but this will be an issue in the future should the KptFileName and KptFileKind variables have different values. * Disable unwrapping to read fnconfig (#2582) * Disable wrapping to read fnconfig * Update index annotation * Update licenses (#2583) - Use go install with explicit tag to avoid modifying go mod - Add licenses to yaml files (new addlicense feature) - Add updated license-check to make all * Revert "update installation doc and Homebrew to use 1.0.0-beta.8 (#2578)" (#2584) This reverts commit 98f0299. * add tests to exercise modify path annotation (#2586) * Update design doc review process (#2594) * Update beta 9 (#2597) * Add missing json annotation to KptFile (#2598) * Implement metrics annotation (#2588) * Implement metrics annotation * Changes * Suggested changes * Suggested changes * Attributor * Design proposal: kpt live -- Inventory to ResourceGroup (#2576) * DD for kpt live apply from STDIN * resolve comments * Update doc with rg fn interaction Co-authored-by: Phani Teja Marupaka <[email protected]> * Add cnrm annotation to merge3 (#2605) * docs: Specify origin in git push command in 3.8 (#2608) Issues: #2596 * upgrade to kyaml 0.13.0+ (#2603) * upgrade to kyaml 0.13.0+ * use the latest kyaml * --image flag typo fix (#2601) Fix typo in `kpt fn eval`'s --image flag description * Simplify and speed up "kpt live" e2e tests (#2613) * Simplify and speed up live e2e tests * Addressed comments * Adding OCI to Kptfile schema Adding OCI to error utils Adding OCI argument parsing to kpt pkg get Adding get/fetch validation Adding pull and untar to support kpt get Making change in md file go fmt and fixing tests * Adds `oci://` to disambiguate `kpt pkg get` argument Factoring upstream fetcher out of switch cases go fmt fixes Ignore scopelint in table tests Adding support for `kpt pkg update` Fix fmt and tests Path must be added to returned absPath Removing git repo error message check * this startup check probably doesn't make sense any more? the pkg update logic doesn't rely on the state of the local git repo to merge Adding OCI support to kpt pkg diff * Mainly involves using upstream.Fetcher methods Adding prototype kpt pkg pull command Based on design and latest conversations Adding kpt pkg push command Minor fixes * Fixing a direct reference to Git field * Adding output to pkg push command Rename upstream pkg to remote Refactoring remote names * Rename Fether to Upstream * Split Upstream and Origin implementations Removing redundant Origin suffix from names Adding compile-time interface assertion go fix formatting Adding missing err checks Fixing package reference case Fixing case sensitive package * Fixing go.sum after rebase * Adjusting merge of update * Adding support for directory in oci subpackage upstream * Fix formatting * Remove check if repo is checked into git for update (#2569) * Remove check if repo is checked into git for update * Update tests * Updating go.sum * Adding previous round of PR feedback * Updating scheme * Adding TODO(oci-support) marker to kpt site content Co-authored-by: Mike Borozdin <[email protected]> Co-authored-by: Natasha Sarkar <[email protected]> Co-authored-by: phani <[email protected]> Co-authored-by: Mengqi Yu <[email protected]> Co-authored-by: Morten Torkildsen <[email protected]> Co-authored-by: Ramon Quitales <[email protected]> Co-authored-by: Karl Isenberg <[email protected]> Co-authored-by: Martin Maly <[email protected]> Co-authored-by: Yuwen Ma <[email protected]> Co-authored-by: Phani Teja Marupaka <[email protected]> Co-authored-by: sdowell <[email protected]> Co-authored-by: Ben Congdon <[email protected]> Co-authored-by: Louis DeJardin <[email protected]>

* design document template (#2563) * updated maintainers * update to the front page * installation paragraph * design doc template * additional process detail * slight updates * update kyaml and cli-utils to latest (#2523) * Do not error while adding merge comment (#2564) * Remove check if repo is checked into git for update (#2569) * Remove check if repo is checked into git for update * Update tests * Update should accept empty path with version (#2572) * Update roadmap for Q4 (#2571) * Update roadmap for Q4 * Suggested changes * Update roadmap (#2574) * Update roadmap * Suggested changes * Update with suggested format * Organize better * update installation doc and Homebrew to use 1.0.0-beta.8 (#2578) * Update kpt book with depends-on annotation (#2565) * Update kpt book with depends-on annotation * Addressed comments * Add snippet for resource dependencies in chapter overview * fix: use KptFileKind variable instead of KptFileName for GVR (#2580) When specifying the Kptfile resource kind, some code were using the KptFileName variable instead of KptFileKind. Both variables currently contain the same value: "Kptfile", but this will be an issue in the future should the KptFileName and KptFileKind variables have different values. * Disable unwrapping to read fnconfig (#2582) * Disable wrapping to read fnconfig * Update index annotation * Update licenses (#2583) - Use go install with explicit tag to avoid modifying go mod - Add licenses to yaml files (new addlicense feature) - Add updated license-check to make all * Revert "update installation doc and Homebrew to use 1.0.0-beta.8 (#2578)" (#2584) This reverts commit 98f0299. * add tests to exercise modify path annotation (#2586) * Update design doc review process (#2594) * Update beta 9 (#2597) * Add missing json annotation to KptFile (#2598) * Implement metrics annotation (#2588) * Implement metrics annotation * Changes * Suggested changes * Suggested changes * Attributor * Design proposal: kpt live -- Inventory to ResourceGroup (#2576) * DD for kpt live apply from STDIN * resolve comments * Update doc with rg fn interaction Co-authored-by: Phani Teja Marupaka <[email protected]> * Add cnrm annotation to merge3 (#2605) * docs: Specify origin in git push command in 3.8 (#2608) Issues: #2596 * upgrade to kyaml 0.13.0+ (#2603) * upgrade to kyaml 0.13.0+ * use the latest kyaml * --image flag typo fix (#2601) Fix typo in `kpt fn eval`'s --image flag description * Simplify and speed up "kpt live" e2e tests (#2613) * Simplify and speed up live e2e tests * Addressed comments * Use reconcile-timeout for live apply tests (#2618) * Fix flaky e2e test (#2622) * Docs: OCI Support design docs (#2589) * Starting to fill in oci design doc * Updates to design doc * Formatting updates * Updating title * Adding why * Update 01-oci-support.md * Update 01-oci-support.md * Docs: Adding yaml examples to kpt pkg update * Docs: add descriptions of pkg pull/push Also adds kpt pkg copy as an alternative * Update 01-oci-support.md Adding comparison of get vs pull * Update 01-oci-support.md Updating the descriptions of `kpt pkg pull` and `kpt pkg push` Co-authored-by: Louis DeJardin <[email protected]> * changed default image pull policy to if-not-present (#2600) * updated installation instructions for kpt 1.0.beta.10 (#2628) * feat: Use fn patch version for cli autocomplete (#2629) This brings the cli function autocompletion in line with other updates to kpt function catalog/documentation, in which we display the fully qualified (patch) version for functions. This is accomplished using the new /catalog-v2.json endpoint which serves an updated schema that includes the latest patch version of each function release. * fix: bash auto completion for CLI flags (#2630) This kpt subcommand is invoked by bash shell completion, which previously returned an error when attempting to perform auto completion for flag values. The actual subcommands and their associated flags are registered with the root Command but not this subcommand, which results in flag validation returning an error for this subcommand. Disabling flag validation fixes this issue and enables auto completion for flags in bash. * Cobra based autocompletion (#2631) * refactor: set klog flags before adding to cmd When flags are set using the Command.Flags().Set() method, the flags are marked as Changed. This results in the Cobra autocompletion logic treating it as though the user passed flags for the Command, ultimately resulting in Cobra not rendering the available subcommands for the root kpt command. This change enables the expected behavior when invoking `kpt <tab>` with the Cobra built-in autocompletion. * refactor: use Cobra built-in autocompletion Prior to this change, the kpt CLI exposed two methods for enabling shell autocompletion: - kpt --install-completion - kpt completion [bash|fish|zsh|powershell] The former was implemented using complete, whereas the latter is provided by Cobra as a default. There is not a clear motivation to offer two methods to provide autocompletion functionality, so this change aims to consolidate these into a single method. This change removes the --install-completion option and applies some fixes to enable proper operation of Cobras autocompletion. The decision was made to favor the Cobra implementation over the custom one in order to reduce maintenance cost and offer similar behavior to other popular CLIs which leverage Cobra (e.g. kubectl) * ci: allow no mozilla repos in create-licenses As of a prior commit there are no dependencies on mozilla repos. This change prevents the script from failing in the case of mozilla_repos being an empty string. * fn render supports function binary executables (#2593) * update base image in Dockerfiles (#2634) * Add annotation reference docs (#2636) * docs: add autocompletion in installation guide (#2635) * Fix go.sum with `go mod tidy` * Fix to optional kpt pkg push argument * Update cli-utils to latest version (#2616) * Remove printers from third_party and use cli-utils (#2627) * docs: fix book hyperlink for depends-on (#2640) The hyperlink was missing a trailing / which was leading to a 404 for the redirected page. * feat: always print truncated function stderr (#2639) * feat: always print truncated function stderr This change makes it so that kpt fn render|eval will always print truncated stderr output from functions. This is intended to improve the experience of debugging functions. * test: add unit test cases for printFnStderr * test: add e2e test for fn eval stderr This test case is intended to check that kpt fn eval outputs the stderr of the function even when the function succeeds. This is doing using a simple function script which emits hello world to stderr. * test: add e2e test for fn render stderr This test case is intended to check that kpt fn render outputs the stderr of the function even when the function succeeds. This is doing using a simple function script which emits hello world to stderr. * Update cli-utils to v0.27.0 (#2644) * Added docs for using exec in fn render (#2642) * Added docs for using exec in fn render * addressed review comment * Removing extra files from merge Co-authored-by: Mike Borozdin <[email protected]> Co-authored-by: Natasha Sarkar <[email protected]> Co-authored-by: phani <[email protected]> Co-authored-by: Mengqi Yu <[email protected]> Co-authored-by: Morten Torkildsen <[email protected]> Co-authored-by: Ramon Quitales <[email protected]> Co-authored-by: Karl Isenberg <[email protected]> Co-authored-by: Martin Maly <[email protected]> Co-authored-by: Yuwen Ma <[email protected]> Co-authored-by: Phani Teja Marupaka <[email protected]> Co-authored-by: sdowell <[email protected]> Co-authored-by: Ben Congdon <[email protected]> Co-authored-by: Louis DeJardin <[email protected]> Co-authored-by: Sunil Arora <[email protected]>

commit c0c9326 Author: Louis DeJardin <[email protected]> Date: Tue Jan 25 23:54:52 2022 +0000 Running go mod tidy and make commit 2681dc4 Merge: aa29f77 3d90e0e Author: Louis DeJardin <[email protected]> Date: Tue Jan 25 23:50:38 2022 +0000 Merge remote-tracking branch 'upstream/main' into HEAD commit aa29f77 Author: Louis DeJardin <[email protected]> Date: Thu Jan 13 15:47:42 2022 -0800 Pull from main into oci-branch (kptdev#2643) * design document template (kptdev#2563) * updated maintainers * update to the front page * installation paragraph * design doc template * additional process detail * slight updates * update kyaml and cli-utils to latest (kptdev#2523) * Do not error while adding merge comment (kptdev#2564) * Remove check if repo is checked into git for update (kptdev#2569) * Remove check if repo is checked into git for update * Update tests * Update should accept empty path with version (kptdev#2572) * Update roadmap for Q4 (kptdev#2571) * Update roadmap for Q4 * Suggested changes * Update roadmap (kptdev#2574) * Update roadmap * Suggested changes * Update with suggested format * Organize better * update installation doc and Homebrew to use 1.0.0-beta.8 (kptdev#2578) * Update kpt book with depends-on annotation (kptdev#2565) * Update kpt book with depends-on annotation * Addressed comments * Add snippet for resource dependencies in chapter overview * fix: use KptFileKind variable instead of KptFileName for GVR (kptdev#2580) When specifying the Kptfile resource kind, some code were using the KptFileName variable instead of KptFileKind. Both variables currently contain the same value: "Kptfile", but this will be an issue in the future should the KptFileName and KptFileKind variables have different values. * Disable unwrapping to read fnconfig (kptdev#2582) * Disable wrapping to read fnconfig * Update index annotation * Update licenses (kptdev#2583) - Use go install with explicit tag to avoid modifying go mod - Add licenses to yaml files (new addlicense feature) - Add updated license-check to make all * Revert "update installation doc and Homebrew to use 1.0.0-beta.8 (kptdev#2578)" (kptdev#2584) This reverts commit 98f0299. * add tests to exercise modify path annotation (kptdev#2586) * Update design doc review process (kptdev#2594) * Update beta 9 (kptdev#2597) * Add missing json annotation to KptFile (kptdev#2598) * Implement metrics annotation (kptdev#2588) * Implement metrics annotation * Changes * Suggested changes * Suggested changes * Attributor * Design proposal: kpt live -- Inventory to ResourceGroup (kptdev#2576) * DD for kpt live apply from STDIN * resolve comments * Update doc with rg fn interaction Co-authored-by: Phani Teja Marupaka <[email protected]> * Add cnrm annotation to merge3 (kptdev#2605) * docs: Specify origin in git push command in 3.8 (kptdev#2608) Issues: kptdev#2596 * upgrade to kyaml 0.13.0+ (kptdev#2603) * upgrade to kyaml 0.13.0+ * use the latest kyaml * --image flag typo fix (kptdev#2601) Fix typo in `kpt fn eval`'s --image flag description * Simplify and speed up "kpt live" e2e tests (kptdev#2613) * Simplify and speed up live e2e tests * Addressed comments * Use reconcile-timeout for live apply tests (kptdev#2618) * Fix flaky e2e test (kptdev#2622) * Docs: OCI Support design docs (kptdev#2589) * Starting to fill in oci design doc * Updates to design doc * Formatting updates * Updating title * Adding why * Update 01-oci-support.md * Update 01-oci-support.md * Docs: Adding yaml examples to kpt pkg update * Docs: add descriptions of pkg pull/push Also adds kpt pkg copy as an alternative * Update 01-oci-support.md Adding comparison of get vs pull * Update 01-oci-support.md Updating the descriptions of `kpt pkg pull` and `kpt pkg push` Co-authored-by: Louis DeJardin <[email protected]> * changed default image pull policy to if-not-present (kptdev#2600) * updated installation instructions for kpt 1.0.beta.10 (kptdev#2628) * feat: Use fn patch version for cli autocomplete (kptdev#2629) This brings the cli function autocompletion in line with other updates to kpt function catalog/documentation, in which we display the fully qualified (patch) version for functions. This is accomplished using the new /catalog-v2.json endpoint which serves an updated schema that includes the latest patch version of each function release. * fix: bash auto completion for CLI flags (kptdev#2630) This kpt subcommand is invoked by bash shell completion, which previously returned an error when attempting to perform auto completion for flag values. The actual subcommands and their associated flags are registered with the root Command but not this subcommand, which results in flag validation returning an error for this subcommand. Disabling flag validation fixes this issue and enables auto completion for flags in bash. * Cobra based autocompletion (kptdev#2631) * refactor: set klog flags before adding to cmd When flags are set using the Command.Flags().Set() method, the flags are marked as Changed. This results in the Cobra autocompletion logic treating it as though the user passed flags for the Command, ultimately resulting in Cobra not rendering the available subcommands for the root kpt command. This change enables the expected behavior when invoking `kpt <tab>` with the Cobra built-in autocompletion. * refactor: use Cobra built-in autocompletion Prior to this change, the kpt CLI exposed two methods for enabling shell autocompletion: - kpt --install-completion - kpt completion [bash|fish|zsh|powershell] The former was implemented using complete, whereas the latter is provided by Cobra as a default. There is not a clear motivation to offer two methods to provide autocompletion functionality, so this change aims to consolidate these into a single method. This change removes the --install-completion option and applies some fixes to enable proper operation of Cobras autocompletion. The decision was made to favor the Cobra implementation over the custom one in order to reduce maintenance cost and offer similar behavior to other popular CLIs which leverage Cobra (e.g. kubectl) * ci: allow no mozilla repos in create-licenses As of a prior commit there are no dependencies on mozilla repos. This change prevents the script from failing in the case of mozilla_repos being an empty string. * fn render supports function binary executables (kptdev#2593) * update base image in Dockerfiles (kptdev#2634) * Add annotation reference docs (kptdev#2636) * docs: add autocompletion in installation guide (kptdev#2635) * Fix go.sum with `go mod tidy` * Fix to optional kpt pkg push argument * Update cli-utils to latest version (kptdev#2616) * Remove printers from third_party and use cli-utils (kptdev#2627) * docs: fix book hyperlink for depends-on (kptdev#2640) The hyperlink was missing a trailing / which was leading to a 404 for the redirected page. * feat: always print truncated function stderr (kptdev#2639) * feat: always print truncated function stderr This change makes it so that kpt fn render|eval will always print truncated stderr output from functions. This is intended to improve the experience of debugging functions. * test: add unit test cases for printFnStderr * test: add e2e test for fn eval stderr This test case is intended to check that kpt fn eval outputs the stderr of the function even when the function succeeds. This is doing using a simple function script which emits hello world to stderr. * test: add e2e test for fn render stderr This test case is intended to check that kpt fn render outputs the stderr of the function even when the function succeeds. This is doing using a simple function script which emits hello world to stderr. * Update cli-utils to v0.27.0 (kptdev#2644) * Added docs for using exec in fn render (kptdev#2642) * Added docs for using exec in fn render * addressed review comment * Removing extra files from merge Co-authored-by: Mike Borozdin <[email protected]> Co-authored-by: Natasha Sarkar <[email protected]> Co-authored-by: phani <[email protected]> Co-authored-by: Mengqi Yu <[email protected]> Co-authored-by: Morten Torkildsen <[email protected]> Co-authored-by: Ramon Quitales <[email protected]> Co-authored-by: Karl Isenberg <[email protected]> Co-authored-by: Martin Maly <[email protected]> Co-authored-by: Yuwen Ma <[email protected]> Co-authored-by: Phani Teja Marupaka <[email protected]> Co-authored-by: sdowell <[email protected]> Co-authored-by: Ben Congdon <[email protected]> Co-authored-by: Louis DeJardin <[email protected]> Co-authored-by: Sunil Arora <[email protected]> commit 45fb5ee Author: Louis DeJardin <[email protected]> Date: Wed Dec 15 20:45:06 2021 -0800 Oci support rebased (kptdev#2621) * design document template (kptdev#2563) * updated maintainers * update to the front page * installation paragraph * design doc template * additional process detail * slight updates * update kyaml and cli-utils to latest (kptdev#2523) * Do not error while adding merge comment (kptdev#2564) * Remove check if repo is checked into git for update (kptdev#2569) * Remove check if repo is checked into git for update * Update tests * Update should accept empty path with version (kptdev#2572) * Update roadmap for Q4 (kptdev#2571) * Update roadmap for Q4 * Suggested changes * Update roadmap (kptdev#2574) * Update roadmap * Suggested changes * Update with suggested format * Organize better * update installation doc and Homebrew to use 1.0.0-beta.8 (kptdev#2578) * Update kpt book with depends-on annotation (kptdev#2565) * Update kpt book with depends-on annotation * Addressed comments * Add snippet for resource dependencies in chapter overview * fix: use KptFileKind variable instead of KptFileName for GVR (kptdev#2580) When specifying the Kptfile resource kind, some code were using the KptFileName variable instead of KptFileKind. Both variables currently contain the same value: "Kptfile", but this will be an issue in the future should the KptFileName and KptFileKind variables have different values. * Disable unwrapping to read fnconfig (kptdev#2582) * Disable wrapping to read fnconfig * Update index annotation * Update licenses (kptdev#2583) - Use go install with explicit tag to avoid modifying go mod - Add licenses to yaml files (new addlicense feature) - Add updated license-check to make all * Revert "update installation doc and Homebrew to use 1.0.0-beta.8 (kptdev#2578)" (kptdev#2584) This reverts commit 98f0299. * add tests to exercise modify path annotation (kptdev#2586) * Update design doc review process (kptdev#2594) * Update beta 9 (kptdev#2597) * Add missing json annotation to KptFile (kptdev#2598) * Implement metrics annotation (kptdev#2588) * Implement metrics annotation * Changes * Suggested changes * Suggested changes * Attributor * Design proposal: kpt live -- Inventory to ResourceGroup (kptdev#2576) * DD for kpt live apply from STDIN * resolve comments * Update doc with rg fn interaction Co-authored-by: Phani Teja Marupaka <[email protected]> * Add cnrm annotation to merge3 (kptdev#2605) * docs: Specify origin in git push command in 3.8 (kptdev#2608) Issues: kptdev#2596 * upgrade to kyaml 0.13.0+ (kptdev#2603) * upgrade to kyaml 0.13.0+ * use the latest kyaml * --image flag typo fix (kptdev#2601) Fix typo in `kpt fn eval`'s --image flag description * Simplify and speed up "kpt live" e2e tests (kptdev#2613) * Simplify and speed up live e2e tests * Addressed comments * Adding OCI to Kptfile schema Adding OCI to error utils Adding OCI argument parsing to kpt pkg get Adding get/fetch validation Adding pull and untar to support kpt get Making change in md file go fmt and fixing tests * Adds `oci://` to disambiguate `kpt pkg get` argument Factoring upstream fetcher out of switch cases go fmt fixes Ignore scopelint in table tests Adding support for `kpt pkg update` Fix fmt and tests Path must be added to returned absPath Removing git repo error message check * this startup check probably doesn't make sense any more? the pkg update logic doesn't rely on the state of the local git repo to merge Adding OCI support to kpt pkg diff * Mainly involves using upstream.Fetcher methods Adding prototype kpt pkg pull command Based on design and latest conversations Adding kpt pkg push command Minor fixes * Fixing a direct reference to Git field * Adding output to pkg push command Rename upstream pkg to remote Refactoring remote names * Rename Fether to Upstream * Split Upstream and Origin implementations Removing redundant Origin suffix from names Adding compile-time interface assertion go fix formatting Adding missing err checks Fixing package reference case Fixing case sensitive package * Fixing go.sum after rebase * Adjusting merge of update * Adding support for directory in oci subpackage upstream * Fix formatting * Remove check if repo is checked into git for update (kptdev#2569) * Remove check if repo is checked into git for update * Update tests * Updating go.sum * Adding previous round of PR feedback * Updating scheme * Adding TODO(oci-support) marker to kpt site content Co-authored-by: Mike Borozdin <[email protected]> Co-authored-by: Natasha Sarkar <[email protected]> Co-authored-by: phani <[email protected]> Co-authored-by: Mengqi Yu <[email protected]> Co-authored-by: Morten Torkildsen <[email protected]> Co-authored-by: Ramon Quitales <[email protected]> Co-authored-by: Karl Isenberg <[email protected]> Co-authored-by: Martin Maly <[email protected]> Co-authored-by: Yuwen Ma <[email protected]> Co-authored-by: Phani Teja Marupaka <[email protected]> Co-authored-by: sdowell <[email protected]> Co-authored-by: Ben Congdon <[email protected]> Co-authored-by: Louis DeJardin <[email protected]>

* design document template (kptdev#2563) * updated maintainers * update to the front page * installation paragraph * design doc template * additional process detail * slight updates * update kyaml and cli-utils to latest (kptdev#2523) * Do not error while adding merge comment (kptdev#2564) * Remove check if repo is checked into git for update (kptdev#2569) * Remove check if repo is checked into git for update * Update tests * Update should accept empty path with version (kptdev#2572) * Update roadmap for Q4 (kptdev#2571) * Update roadmap for Q4 * Suggested changes * Update roadmap (kptdev#2574) * Update roadmap * Suggested changes * Update with suggested format * Organize better * update installation doc and Homebrew to use 1.0.0-beta.8 (kptdev#2578) * Update kpt book with depends-on annotation (kptdev#2565) * Update kpt book with depends-on annotation * Addressed comments * Add snippet for resource dependencies in chapter overview * fix: use KptFileKind variable instead of KptFileName for GVR (kptdev#2580) When specifying the Kptfile resource kind, some code were using the KptFileName variable instead of KptFileKind. Both variables currently contain the same value: "Kptfile", but this will be an issue in the future should the KptFileName and KptFileKind variables have different values. * Disable unwrapping to read fnconfig (kptdev#2582) * Disable wrapping to read fnconfig * Update index annotation * Update licenses (kptdev#2583) - Use go install with explicit tag to avoid modifying go mod - Add licenses to yaml files (new addlicense feature) - Add updated license-check to make all * Revert "update installation doc and Homebrew to use 1.0.0-beta.8 (kptdev#2578)" (kptdev#2584) This reverts commit 98f0299. * add tests to exercise modify path annotation (kptdev#2586) * Update design doc review process (kptdev#2594) * Update beta 9 (kptdev#2597) * Add missing json annotation to KptFile (kptdev#2598) * Implement metrics annotation (kptdev#2588) * Implement metrics annotation * Changes * Suggested changes * Suggested changes * Attributor * Design proposal: kpt live -- Inventory to ResourceGroup (kptdev#2576) * DD for kpt live apply from STDIN * resolve comments * Update doc with rg fn interaction Co-authored-by: Phani Teja Marupaka <[email protected]> * Add cnrm annotation to merge3 (kptdev#2605) * docs: Specify origin in git push command in 3.8 (kptdev#2608) Issues: kptdev#2596 * upgrade to kyaml 0.13.0+ (kptdev#2603) * upgrade to kyaml 0.13.0+ * use the latest kyaml * --image flag typo fix (kptdev#2601) Fix typo in `kpt fn eval`'s --image flag description * Simplify and speed up "kpt live" e2e tests (kptdev#2613) * Simplify and speed up live e2e tests * Addressed comments * Adding OCI to Kptfile schema Adding OCI to error utils Adding OCI argument parsing to kpt pkg get Adding get/fetch validation Adding pull and untar to support kpt get Making change in md file go fmt and fixing tests * Adds `oci://` to disambiguate `kpt pkg get` argument Factoring upstream fetcher out of switch cases go fmt fixes Ignore scopelint in table tests Adding support for `kpt pkg update` Fix fmt and tests Path must be added to returned absPath Removing git repo error message check * this startup check probably doesn't make sense any more? the pkg update logic doesn't rely on the state of the local git repo to merge Adding OCI support to kpt pkg diff * Mainly involves using upstream.Fetcher methods Adding prototype kpt pkg pull command Based on design and latest conversations Adding kpt pkg push command Minor fixes * Fixing a direct reference to Git field * Adding output to pkg push command Rename upstream pkg to remote Refactoring remote names * Rename Fether to Upstream * Split Upstream and Origin implementations Removing redundant Origin suffix from names Adding compile-time interface assertion go fix formatting Adding missing err checks Fixing package reference case Fixing case sensitive package * Fixing go.sum after rebase * Adjusting merge of update * Adding support for directory in oci subpackage upstream * Fix formatting * Remove check if repo is checked into git for update (kptdev#2569) * Remove check if repo is checked into git for update * Update tests * Updating go.sum * Adding previous round of PR feedback * Updating scheme * Adding TODO(oci-support) marker to kpt site content Co-authored-by: Mike Borozdin <[email protected]> Co-authored-by: Natasha Sarkar <[email protected]> Co-authored-by: phani <[email protected]> Co-authored-by: Mengqi Yu <[email protected]> Co-authored-by: Morten Torkildsen <[email protected]> Co-authored-by: Ramon Quitales <[email protected]> Co-authored-by: Karl Isenberg <[email protected]> Co-authored-by: Martin Maly <[email protected]> Co-authored-by: Yuwen Ma <[email protected]> Co-authored-by: Phani Teja Marupaka <[email protected]> Co-authored-by: sdowell <[email protected]> Co-authored-by: Ben Congdon <[email protected]> Co-authored-by: Louis DeJardin <[email protected]>

* design document template (kptdev#2563) * updated maintainers * update to the front page * installation paragraph * design doc template * additional process detail * slight updates * update kyaml and cli-utils to latest (kptdev#2523) * Do not error while adding merge comment (kptdev#2564) * Remove check if repo is checked into git for update (kptdev#2569) * Remove check if repo is checked into git for update * Update tests * Update should accept empty path with version (kptdev#2572) * Update roadmap for Q4 (kptdev#2571) * Update roadmap for Q4 * Suggested changes * Update roadmap (kptdev#2574) * Update roadmap * Suggested changes * Update with suggested format * Organize better * update installation doc and Homebrew to use 1.0.0-beta.8 (kptdev#2578) * Update kpt book with depends-on annotation (kptdev#2565) * Update kpt book with depends-on annotation * Addressed comments * Add snippet for resource dependencies in chapter overview * fix: use KptFileKind variable instead of KptFileName for GVR (kptdev#2580) When specifying the Kptfile resource kind, some code were using the KptFileName variable instead of KptFileKind. Both variables currently contain the same value: "Kptfile", but this will be an issue in the future should the KptFileName and KptFileKind variables have different values. * Disable unwrapping to read fnconfig (kptdev#2582) * Disable wrapping to read fnconfig * Update index annotation * Update licenses (kptdev#2583) - Use go install with explicit tag to avoid modifying go mod - Add licenses to yaml files (new addlicense feature) - Add updated license-check to make all * Revert "update installation doc and Homebrew to use 1.0.0-beta.8 (kptdev#2578)" (kptdev#2584) This reverts commit 98f0299. * add tests to exercise modify path annotation (kptdev#2586) * Update design doc review process (kptdev#2594) * Update beta 9 (kptdev#2597) * Add missing json annotation to KptFile (kptdev#2598) * Implement metrics annotation (kptdev#2588) * Implement metrics annotation * Changes * Suggested changes * Suggested changes * Attributor * Design proposal: kpt live -- Inventory to ResourceGroup (kptdev#2576) * DD for kpt live apply from STDIN * resolve comments * Update doc with rg fn interaction Co-authored-by: Phani Teja Marupaka <[email protected]> * Add cnrm annotation to merge3 (kptdev#2605) * docs: Specify origin in git push command in 3.8 (kptdev#2608) Issues: kptdev#2596 * upgrade to kyaml 0.13.0+ (kptdev#2603) * upgrade to kyaml 0.13.0+ * use the latest kyaml * --image flag typo fix (kptdev#2601) Fix typo in `kpt fn eval`'s --image flag description * Simplify and speed up "kpt live" e2e tests (kptdev#2613) * Simplify and speed up live e2e tests * Addressed comments * Use reconcile-timeout for live apply tests (kptdev#2618) * Fix flaky e2e test (kptdev#2622) * Docs: OCI Support design docs (kptdev#2589) * Starting to fill in oci design doc * Updates to design doc * Formatting updates * Updating title * Adding why * Update 01-oci-support.md * Update 01-oci-support.md * Docs: Adding yaml examples to kpt pkg update * Docs: add descriptions of pkg pull/push Also adds kpt pkg copy as an alternative * Update 01-oci-support.md Adding comparison of get vs pull * Update 01-oci-support.md Updating the descriptions of `kpt pkg pull` and `kpt pkg push` Co-authored-by: Louis DeJardin <[email protected]> * changed default image pull policy to if-not-present (kptdev#2600) * updated installation instructions for kpt 1.0.beta.10 (kptdev#2628) * feat: Use fn patch version for cli autocomplete (kptdev#2629) This brings the cli function autocompletion in line with other updates to kpt function catalog/documentation, in which we display the fully qualified (patch) version for functions. This is accomplished using the new /catalog-v2.json endpoint which serves an updated schema that includes the latest patch version of each function release. * fix: bash auto completion for CLI flags (kptdev#2630) This kpt subcommand is invoked by bash shell completion, which previously returned an error when attempting to perform auto completion for flag values. The actual subcommands and their associated flags are registered with the root Command but not this subcommand, which results in flag validation returning an error for this subcommand. Disabling flag validation fixes this issue and enables auto completion for flags in bash. * Cobra based autocompletion (kptdev#2631) * refactor: set klog flags before adding to cmd When flags are set using the Command.Flags().Set() method, the flags are marked as Changed. This results in the Cobra autocompletion logic treating it as though the user passed flags for the Command, ultimately resulting in Cobra not rendering the available subcommands for the root kpt command. This change enables the expected behavior when invoking `kpt <tab>` with the Cobra built-in autocompletion. * refactor: use Cobra built-in autocompletion Prior to this change, the kpt CLI exposed two methods for enabling shell autocompletion: - kpt --install-completion - kpt completion [bash|fish|zsh|powershell] The former was implemented using complete, whereas the latter is provided by Cobra as a default. There is not a clear motivation to offer two methods to provide autocompletion functionality, so this change aims to consolidate these into a single method. This change removes the --install-completion option and applies some fixes to enable proper operation of Cobras autocompletion. The decision was made to favor the Cobra implementation over the custom one in order to reduce maintenance cost and offer similar behavior to other popular CLIs which leverage Cobra (e.g. kubectl) * ci: allow no mozilla repos in create-licenses As of a prior commit there are no dependencies on mozilla repos. This change prevents the script from failing in the case of mozilla_repos being an empty string. * fn render supports function binary executables (kptdev#2593) * update base image in Dockerfiles (kptdev#2634) * Add annotation reference docs (kptdev#2636) * docs: add autocompletion in installation guide (kptdev#2635) * Fix go.sum with `go mod tidy` * Fix to optional kpt pkg push argument * Update cli-utils to latest version (kptdev#2616) * Remove printers from third_party and use cli-utils (kptdev#2627) * docs: fix book hyperlink for depends-on (kptdev#2640) The hyperlink was missing a trailing / which was leading to a 404 for the redirected page. * feat: always print truncated function stderr (kptdev#2639) * feat: always print truncated function stderr This change makes it so that kpt fn render|eval will always print truncated stderr output from functions. This is intended to improve the experience of debugging functions. * test: add unit test cases for printFnStderr * test: add e2e test for fn eval stderr This test case is intended to check that kpt fn eval outputs the stderr of the function even when the function succeeds. This is doing using a simple function script which emits hello world to stderr. * test: add e2e test for fn render stderr This test case is intended to check that kpt fn render outputs the stderr of the function even when the function succeeds. This is doing using a simple function script which emits hello world to stderr. * Update cli-utils to v0.27.0 (kptdev#2644) * Added docs for using exec in fn render (kptdev#2642) * Added docs for using exec in fn render * addressed review comment * Removing extra files from merge Co-authored-by: Mike Borozdin <[email protected]> Co-authored-by: Natasha Sarkar <[email protected]> Co-authored-by: phani <[email protected]> Co-authored-by: Mengqi Yu <[email protected]> Co-authored-by: Morten Torkildsen <[email protected]> Co-authored-by: Ramon Quitales <[email protected]> Co-authored-by: Karl Isenberg <[email protected]> Co-authored-by: Martin Maly <[email protected]> Co-authored-by: Yuwen Ma <[email protected]> Co-authored-by: Phani Teja Marupaka <[email protected]> Co-authored-by: sdowell <[email protected]> Co-authored-by: Ben Congdon <[email protected]> Co-authored-by: Louis DeJardin <[email protected]> Co-authored-by: Sunil Arora <[email protected]>

DD for kpt live apply from STDIN

2ca5725

yuwenma requested review from droot and mikebz November 6, 2021 00:50

mikebz added the design-doc label Nov 8, 2021

mikebz mentioned this pull request Nov 9, 2021

[Question] clarifications on live commands relationship with kustomize #1545

Closed

mikebz linked an issue Nov 9, 2021 that may be closed by this pull request

[Question] clarifications on live commands relationship with kustomize #1545

Closed

mikebz reviewed Nov 9, 2021

View reviewed changes

docs/design-docs/01-kpt-live-stdin.md Outdated Show resolved Hide resolved

mikebz reviewed Nov 9, 2021

View reviewed changes

phanimarupaka reviewed Nov 11, 2021

View reviewed changes

docs/design-docs/01-kpt-live-stdin.md Outdated Show resolved Hide resolved

docs/design-docs/01-kpt-live-stdin.md Outdated Show resolved Hide resolved

docs/design-docs/01-kpt-live-stdin.md Outdated Show resolved Hide resolved

marshall007 reviewed Nov 11, 2021

View reviewed changes

docs/design-docs/01-kpt-live-stdin.md Outdated Show resolved Hide resolved

droot reviewed Nov 11, 2021

View reviewed changes

yuwenma requested review from mikebz, droot and phanimarupaka November 17, 2021 23:07

yuwenma force-pushed the kpt-actuation-dd branch from 82793bc to 02030be Compare November 18, 2021 05:13

yuwenma changed the title ~~Design proposal: kpt live apply from STDIN~~ Design proposal: kpt live -- Inventory to ResourceGroup Nov 18, 2021