Platform Stability & Engines Optimization

Check the Components during continuous operations.

There are various components which start degrading over time during the continuous operations. These degraded components start posing stabilities issues. The organization or team requires to identify all measurable components in the platform. 

Check the Components which break during scaling up

Big Data works in cluster mode in real time with huge load. Some time resource manager does not comply the rules set and get corrupted and hence corrupts the node which further HDFS file.

Optimize and Troubleshoot Bigdata jobs.

Big Data jobs comprise of queries run with specific tasks, either run manually or by schedulers. Big Data jobs require suitable and compatible compute engines hardware with tuned parameters. These parameters are from Linux Machine or Virtual Box and from Big Data - Hadoop or Hive or Spark or Airflow or Trino or other similar. 

Tune Bigdata engines with custom utilities (e.g. Sparklens for Spark).

There is further custom code to tune Big Data engines. For example, Sparklens is an open source Spark profiling tool used with Spark application. Sparklens helps in tuning Spark applications by identifying the potential opportunities for optimizations with respect to driver side computations, lack of parallelism, skew, etc. (Details about Sparklens from Qubole - https://github.com/qubole/sparklens).