Metaflow Configs (Netflix#1962)

* Make CLI lazy load

This will allow parameters to be added when processing the `start` step
(for the upcomming config change).

* Try to fix tests

* Support lazy CLI in runner

* Stub fixes

* Change top-level flow decorator options to be prefixed by METAFLOW_FLOW_

Previously, options like `branch` and `name` (injected by the project
decorator for example) could be set using `METAFLOW_BRANCH`. They now
need to be set using `METAFLOW_FLOW_BRANCH`.

This change is made to prevent clashes between regular metaflow
configuration settings and decorator level options.

No other changes are made so `METAFLOW_RUN_MAX_WORKERS` still works
as expected and `METAFLOW_PYLINT` as well.

* Initial Config object

* Move from --<configname> <config> to --config <configname> <config>

* Fix runner use of configs

* Multiple fix plus sample flow

Several fixes:
  - fixed an issue with default values
  - better handling of parameter defaults as configs
  - handle config defaults as functions
  - ConfigValue is more "dict"-like
  - made <myflow>.configs and <myflow>.steps work properly
  - renamed resolve_configs to init

* Addressed comments. Added more documentation/explanation

Specifically:
  - moved things out of the INFO file
  - added to_dict
  - renamed user_configs to config_parameters

* Added test, more cleanup

Specifically:
  - made config values immutable
  - cleaned up state stored in FlowSpec
  - added a test exercising configs in various places

* Fixup conda decorator

* Fix parallel tests

* Fix current singleton test (conflict with `steps`)

* Call decorator init method on non-static decorators

* Several fixes

 - Separate out value and file (so default and default_value and --config and --config-value)
 - Provide classes for step and flow config decorators with proxy objects
 - Split things into several files (it was getting too long)
 - Addressed all bugs discussed

* Better handling of default options

* Changed names

* Fix includefile

* Remove more old code

* Bug fixes and better Custom*Decorator behavior.

Fixed some typos and updated test to reflect latest code.

Fixed a few other issues:
  - fixed an issue where a config was used in different decorators
    causing it to produce an incorrect access string
  - made the decorators work with or without arguments

* do not map config parameters to CLI command for argo/step functions

* Added string parsers. Added test for mutable flow/step. Added debug messages

Use METAFLOW_DEBUG_USERCONF=1 to get a bit more detail.

Should be feature complete now.

* Call param init even when no config present

* Reset cached parameters properly

* Updated tests

* More tests and a few tweaks

* Fix test

* Trigger tests

* Fix no command case; fix deployer

Author: romain-intel

README.md

@@ -67,3 +67,4 @@ There are several ways to get in touch with us:
 
 ## Contributing
 We welcome contributions to Metaflow. Please see our [contribution guide](https://docs.metaflow.org/introduction/contributing-to-metaflow) for more details.
+

metaflow/__init__.py

@@ -103,6 +103,9 @@ class and related decorators.
 
 from .parameters import Parameter, JSONTypeClass, JSONType
 
+from .user_configs.config_parameters import Config, config_expr
+from .user_configs.config_decorators import CustomFlowDecorator, CustomStepDecorator
+
 # data layer
 # For historical reasons, we make metaflow.plugins.datatools accessible as
 # metaflow.datatools. S3 is also a tool that has historically been available at the