Our Methodologies

Almost all organization reaches to Big Data Maturity level during its journey of operation. The organization which implements Big Data first time, does assessment first. There are various frameworks used to assess. These frameworks are KDD, CRISP-DM, SEMMA, OSEMN, and TDSP. Out of these, CRISP-DM is widely used. While TDSP – Team Data Science Process is new and recently developed by Microsoft.

Challenges at Organization 

The organization, which is already practicing Big Data from some time or long time, faces different set of challenges. The organization is not able to encash the value locked away within that data despite of massive growth of data. There are a number of underlying cascaded challenges which prevents to encash the values. The biggest fact is Data will remain distributed, stored all over the place, and will appear in more and more systems.

• There are various different data silos inside the organization.
• The data is distributed across various silos, and some of them cannot even be queried at the necessary performance for organizational analytics needs.
• Reach into the data silos, pull aspects out of them, and then combine them in one location.
• Make Data Warehouse out of various data silos.
• Sometimes it is not known where data is even available to use, and only tribal knowledge in the company, or years of experience with internal setups, can help you find the right data.
• The data the organization specifically need today is not in the data warehouse generally.
• Getting the data added is a painful and expensive process full of hurdles.
• Querying the various source databases requires different connections along with different queries running different SQL dialects.
• They look similar enough but they behave just differently enough to cause confusion and the need to learn the details.
• The organizations end up only scratching the surface of data and not gaining the signifacnt understanding.
• Overall but the fact is Data will remain distributed, stored all over the place, and will appear in more and more systems.
• Understanding the data is very crucial for improving the business.
• The organization requires to use different tools and technologies to query and work with the data.
• There is lack of a good way to query the data or to optimize the query.
• The federated SQL query is required to query different systems for complex tasks with huge learning curve within the organization.
• The organization will require faster response time along with better insights out of entire data.
• The complex queries execution on massive data sets runs into limitations.
• The organization will require powerful compute power along with compatible storage system otherwise getting output of query may take multiple days.
• The organization has to trade-off between using of traditional data warehouse or recently developed data lake considering cost and complexities.
• Even the approach of a data lakehouse that promises to combine the advantages of warehouses and lakes will not be the sole solution for the organization under certain circumstances.
• The organization requires competitive tools to visualize and analyze and these tools are not easily available. And there are mounting challenge on how to use.
• Finally the organization requires suitable deployment and runtime platforms to fulfill day to day business needs.

Our Approach 

Find Technical specification (identification and representation) of data, representing individual business unit. How this technical specification is configured in entire data ecosystem of the organization along with others considering boundaries and layers in between.  How this technical specification of individual business unit is interacting with the rest of data ecosystem.

Technical data specification of individual business unit can be identified with the following set of questionnaire. 

• Is data stored on cloud or on premise ?
• How diverse storage mechanisms are available for data ?
  • Relational databases
  • NoSQL databases
  • Document databases
  • Key-value stores
  • Object storage systems
    • Amazon S3
    • Microsoft Azure Blob Storage
    • Google Cloud Storage
    • S3-compatible storage
• How expensive the traditional approach of creating and maintaining large, dedicated data warehouses in organizations.
• Whether traditional Hive or Data Lake or mix of both.
• Are there standard tools to allow users to query and inspect the data in all these different systems.
• Are general-purpose relational database that serves to replace Microsoft SQL Server, Oracle Database, MySQL, or PostgreSQL.
• Are other databases designed and optimized for data warehousing or analytics, such as Teradata, Netezza, Vertica, and Amazon Redshift.
• Whether OLAP or OLTP ?
• Whether SQL or No-SQL
• Do we have tools around SQL
• Only SQL like queries will be enough?
• Whether will require federated or distributed SQL to query HDFS or Object storage or RDBMS.
• Are different query languages and analysis tools available to query NoSQL or other data formats.
• Will defined and formulated framework work for entire data silos

In addition to identifying Technical Specification, there is a requirement of Efficient Techniques. Efficient techniques include , Parallel processing and heavy optimizations regularly lead to performance improvements for your analysis.

• In-memory parallel processing
• Pipelined execution across nodes in the cluster
• A multithreaded execution model to keep all the CPU cores busy
• Efficient flat-memory data structures to minimize Java garbage collection
• Java bytecode generation
• In addition to above, there are features in every Big Data engines, those need to be tuned.