Account Creation and Configuration

Configure Product Account with IAM Roles or similar methodologies

User require to create IAM Role or similar component at clouds (AWS or Azure or GCP) with appropriate policies. Big Data organization like Databricks or Qubole or Others use created role to provision cloud infrastructure for the initial set up and storage. These steps may vary based upon the organization and organization. Once initial set up is completed, user can create cluster (Hadoop or Hive or Spark or Airflow or Presto) or compute engines (SQL or Other). User can use Clusters or Compute engines for the various business purposes.

Configure Storage for the account

Object storage is created in the respective cloud. AWS S3, or Azure Blob Storage or Google Cloud Storage is created based on individual use case. Absolute path of the object storage is used while provisioning product plan account. HDFS storage is also created and configured if object storage is either not required or applicable. HDFS storage details is used while configuring Hadoop or Hive or Spark or Airflow or Presto or any other or Data Lake.

Configure VPC or similar infrastructure

Public or Private Virtual Private Cloud (AWS or GCP) or Azure Virtual Network is created within the organization to make fence with defined CIDR.

Configure or Migrate Custom MetaStore

External Metastore is created for Hive. Metastore could be of MySql or Postgress or Oracle. Migrating of Metastore could be from embedded derby to external or from MySql to Postgress or vice versa. Migrating of Metastore could be migrating database from lower version to higher version.

Troubleshoot Account Configuration

There are various scenario's where account configuration is troubleshooted. Fixing user credentials that is used for account creation. Fixing Role or Key or Token or combination of any of these. Fixing Storage permission or policies. Fixing other dependencies.  

Configure custom Tunnel between Control and Data Planes

Databricks or Qubole or Organization with similar set up work on the concept of Control Plane and Data Plane. Control Plane is lying in the organization territory with access while Data Plane is the individual customer territory. Customer uses Data Plane for different technical or business purposes. There is tunnel between Control Plane and Data Plane. Control Plane and Data Plane communicate with each other through Tunnel configured as bastion node (another compute machine). There is thrift concept utilized while creating and configuring the tunnel or tunnel server. 

Identifying whether project account will be created through UI or API

There are two ways to create project account, either through UI or API. Customer Admin user can use Control Plane UI to create new account. Organization in general provides various API for different purposes. User can use the account creation API with required fields values in JSON payload. Both ways project account is created, in case any issues appear, these are troubleshooted.