TDP getting started and prerequisites

[rpignolet]

Context

• tdp-collection is a repository which is an Ansible collection to deploy and configure an Hadoop cluster.
• tdp-collection-extras is a repository which is an Ansible collection to deploy and configure extras service like Livy and Airflow but have roles to deploy some TDP prerequisites like Kerberos and LDAP.
• tdp-getting-started is a repository with multiple Bash script and Ansible playbooks to setup VMs with vagrant, configure VMs with TDP prerequisites, deploy TDP with custom playbooks (for each service a playbook) and deploy TDP extras (for each service a playbook).

Problem

tdp-getting-started is used by users to deploy TDP out of Vagrant like AWS or custom VMs solution or believe this is the production solution. This is not the goal of tdp-getting-started which is a "sandbox" to play with TDP like Minikube for Kubernetes or Devstack for Openstack. When you want to deploy Kubernetes yourself on AWS (without managed service), you do not deploy Minikube but a real Kubernetes.

Proposition

1. The repository tdp-getting-started should be renamed tdp-sandbox or minitdp or devtdp.

2. We need to keep somewhere a tdp-getting-started but it should be documentation or a skeleton without the creation of VMs. If this is a skeleton, the tdp-sandbox can use the skeleton and just add at the beginning the VMs creation with Vagrant. For exemple, when you want to used the tdp-sandbox, the setup script clone the tdp-getting-started skeleton, launch the VMs with Vagrant and launch playbooks inside the tdp-getting-started skeleton. This way the tdp-getting-started skeleton can be a bootstrap for deploying a TDP production cluster.

3. We should create a repository tdp-collection-prerequisites which is an Ansible collection to deploy all prerequisites for a TDP cluster like Kerberos, LDAP server, OS configuration like ulimit, ... and keep tdp-collection-extras only for extra services like Livy, Airflow, ... This prerequisites repository can be used as is in the tdp-sandbox. The tdp-sandbox should have only 5 playbooks : setup.yml (checkout git submodule, download TDP binaries, etc.), vagrant.yml, tdp-prerequisites.yml, tdp-collection-all.yml and tdp-collection-extras-all.yml.

[Nuttymoon]

Agreed that tdp-getting-started is confusing at the moment.

As discussed this afternoon I would suggest that:

• tdp-getting-started stays the main entry point for new users
• Vagrant and Ansible inventory generation related playbooks should be moved to a collection tdp-collection-sandbox. This collection could offer to target other types of infrastructure in the future (e.g. Cloud VMs clusters via terraform)
• Prerequisites are moved to tdp-collection-prerequisites, including the download of TDP binaries
• tdp-getting-started contains scripts and documentation to setup an Ansible environment using all the collections. Ideally tdp-lib will be used to deploy the components.

[ChristopheLRTE]

It sounds good !
Are all these repo aimed to be maintained in the time ?

[Nuttymoon]

Yes of course @ChristopheLRTE

[rpignolet]

If a "sandbox" is just one VM with TDP pre-deployed, we should not used this name for the collection which deploy VM with vagrant. tdp-collection-vagrant ?

[wdavidw]

Agreed for the -vagrant, we get what it is. Alternative would be -minicluster, 7vms, -xs|-s|-m|....

I personally don't like the collection term, my opinion is that it provides no information. I even find it confusing. Here are some suggestions: tdp-install, tdp-deploy-, tdp-cluster- or tdp-env- (the choice in ryba) are more clear to me.

[DanielJohnHarty]

Your ideas are great - I have a few thoughts (mainly about naming):

1. I think that tdp-sandbox is a good name as it is clear and mirrors the naming used by Hortonworks for their demo deployment. Plus it's likely that the majority of users evaluating TDP are doing so to replace Hortonworks now that it's behind a paywall.

2. I like this idea but I think the naming is crucial to avoid replacing one confusion for another, so I would describe it as a something along the lines of a tdp-deployment-template. I think that the natural evolution of this idea would lead to multiple tdp-deployment-templates with different components in different configurations e.g. tdp-deployment-template-core and tdp-deployment-template-datascience. I think that tdp-deployment-templates should be in their own repositories to manage their evolution individually.

3. tdp-collection-prerequisites is good idea but a more generic name like tdp-deployment-prerequisites is less likely to become misleading in the future as collections are added, removed or renamed.

A proposal from me
Deploying a tdp-getting-started cluster performs a small set of crude integration tests (test-cluster-deployment.yml). As far as I know these are the only cross component integration tests performed on TDP. This means that an update to a collection can (and often has) broken tdp-getting-started or has had some secondary effects meaning it deploys but doesn't behave as expected. I propose that that tdp-getting-started gets a wider more thorough set of integration tests and that it is used not just as a sandbox but also to test that collection changes are working as expected on a deployment, with a daily scheduled deployment (at least). If we move forward with the tdp-deployment-template idea, these tests would naturally evolve with the template.

[rpignolet]

Yes the word "sandbox" should be used with care because, other sandbox like Minikube have a pre-deployed cluster inside one VM (or maybe container). Our current "getting started" spawn multiple VM instead of one and deploy the TDP cluster with the tdp-collection instead of providing an already deployed TDP cluster.

[DanielJohnHarty]

It doesn't exactly roll off the tongue but tdp-deploy-virtual-cluster is quite precise.

[Nuttymoon]

@DanielJohnHarty so tdp-deployment-template would replace tdp-getting-started right? I am not sure that it should be split into multiple repos. We could offer different templates within the same repo to initialize the environment (e.g. bin/init.sh --template data-engineering --infra vagrant).

Agreed that the name tdp-sandbox can be confusing. But I cannot come up with something else.

[rpignolet]

I'm not convinced by the tdp-deployment-template because the only thing that changes is the inventory and we can provide inventory sample inside tdp-getting-started.

[wdavidw]

bin/init.sh --template data-engineering --infra vagrant

At some point yes, a cli asking which targeted deployment platform and profile would be great.

[leopaul36]

Picking from @rpignolet, @Nuttymoon and @DanielJohnHarty, maybe for now we can:

• Rename tdp-getting-started to tdp-cluster-vagrant
• Move Vagrant and Ansible inventory generation related playbooks to tdp-collection-vagrant
• Move the prerequisites to a tdp-collection-prerequisites

In the future we could imagine having:

• tdp-aws / tdp-collection-aws
• tdp-gcp / tdp-collection-gcp
• tdp-azure / tdp-collection-azure
• tdp-lxd / tdp-collection-lxd

All of these being properly referenced in a real Getting started documentation.

Or a unique tdp-deploy / tdp-collection combo featuring subroles for each environment.

I think we should go with the 1st option for now, we can always merge all the different envs / collections into a single one later.

[rpignolet]

Why rename tdp-getting-started to tdp-cluster-vagrant ? It can be use to deploy on AWS if you use the tdp-collection-aws instead of tdp-collection-vagrant. Files in tdp-getting-started like the ansible.cfg or the inventory/topology.ini will not change if you deploy on Vagrant or AWS. I think the goal of tdp-getting-started repository is to have common files to all environment, vagrant, dev on Azure, prod on AWS, etc. Users just have to clone the tdp-getting-started and modify it for their environments.

[mehdibn]

Indeed, I think that we must have a generic project, indecently of the platform. Then, we can have some examples per provider =>The platform should be agnostic from an infra point of view.

[gonzaloetjo]

An other option could be to separate infra and deployment. Vagrant and other cluster environments would go into a tdp-collection-infra with a vagrant-cluster role, and other possible cluster environments in the future. The 'getting-started' deployment would be done through tdp-cli using specific roles from tdp-infra-collection and tdp-prerequisites. A third directory, say tdp-deploy or tdp-working-dir, would state the ansible.cfg, inventory/topology.ini and include a deployment of history db. To summarize:

• tdp-collection-infra: Different environment setups. Includes vagrant-cluster role.
• tdp-prerequisites: Kerberos, LDAP, etc
• tdp-working-dir:
  • tdp-cli to initialize the local tdp-working-dir with a deployment history db (sqlite by default).
  • tdp-cli to deploy TDP to the target nodes with the components they want.
  • The variables for the deployed components are created in individual version controlled local repos.

What I'm not entirely sure is where would architecture design, meaning host selection (edge, master. workers) and posterior provisioning & configuration (python, ntpd) take place. It could be on tdp-infra-collection or on it's own collection.

[wdavidw]

1. I strongly disagree with the naming of prerequisites, Kerberos and LDAP are components of their own, there are required by other components, just like zookeeper is for HDFS and HDFS is for HBase.
2. Let's first start with a tdp-getting-started that works and that works very well, including with tdp-cli. It might already be the case, or not
3. I agree with @mehdibn, one generic project would be nice. Assuming my second point is achieved on one target platform, it could be extended with options (CLI arguments and prompt questions) to target various platforms/deployment scenarios.

[leopaul36]

The following issues have been created to track the changes discussed here:

• Add virtualenv instructions tdp-getting-started#151
• Move the Vagrant related tasks to its own repo tdp-collection-extras#52
• Add a sample Ansible inventory hosts.ini tdp-getting-started#150
• Move the pre-requisites tasks to a tdp-collection-prerequisites tdp-collection-extras#53
• Add tdp-lib for deploying TDP tdp-getting-started#152
• Add an option to setup script to enable feature tdp-getting-started#153