How the Associative Information System Solves Big Data and Related Business Problems
Until now, all solutions addressing database management involve some form of RDBMS solution, highly laden with the overhead of “structure.” This inherently creates many limitations and frustrations in attempting to manage Big and Complex Data. The basic problem is RDBMS was not created to handle the vast amounts and varieties of data we have today. The new crop of systems attempting to solve this dilemma also have numerous structural and mathematical limitations on their ability to function and scale, as well as such complexity it calls for sophisticated and costly trained IT management staff and numerous servers to attempt to handle it.
Associative technology, specifically the patented w solution, as we will see, removes the structural limitations and numerous requirements of today for solving this issue and related problems are also resolved. This is the evolution of data science to reach well beyond today’s systems that exist in the third normal form-all the way past the 4th, 5th and 6th to the “N’ normal form–an exponential leap in science and technology to never before seen efficiency, performance, scalability and security. In fact, it is not simply a database but it is much more, it is an Associative Information System. It involves an object oriented Vector Based Pointer System. We have moved from “Big Data” to “Big Information.” This article will present a general overview of the technology as a deep dive requires more time and space.
This associative database platform is 100 percent binary and compatible with all heterogeneous systems, and inherently de-duplicates all data into single-instance-storage. No need to normalize data structures, worry about data bloat, or maintain large farms of disk arrays; with single-instance-storage technology, the same information can be stored, with no compression or loss of fidelity, while decreasing typical storage requirements by a factor of 6 to 1 over typical Big Data solutions. This approach to data storage dramatically improves performance and ease of administration. Quickly scale Big Data without concern for data redundancy and lack of storage efficiency, allowing users to store and process thousands of petabytes on one single instance.
The basis of the technology is an ‘n’-dimensional associative memory system and solution development platform which can be utilized as a high-performance information store for any data set aggregation, without any design phase, to target complex information system analytics and be customized at run time to meet evolving requirements. The ability to rapidly build and modify any information system or data model (as a knowledge representation) and have it able to be evolved and adapted over time, with minimal or no programming, is key to successful implementation of any modern information system.
The Associative Technology does not require or make use of any Database Management System (RDBMS) to achieve its capabilities. It is a new methodology for storing and managing all levels and complexities of relationships. The Technology is scalable and yet is light enough to run on even the most minimal of sensors and smart phones.
Large organizations globally face the following problems with their IT systems:
• Cost - $600+ Billion spent annually in IT
• Success Rates-more than 80 percent of IT Projects either fail or overrun schedules and budget significantly (>2x). Projects that do get delivered typically have less than 50 percent of their original functional specification
• Inflexibility–Changes to existing systems and integration with other systems is extremely difficult
As a result, the IT systems that support the business wind up slowing its evolution. Up-to-date, consolidated or composite views of the business across systems are still largely unavailable, even the largest, best-financed corporations.
• Difficulty with Providing Information: Any new questions that were not foreseen when the system was originally designed become backlogged change orders with turn-around times of days, weeks, or months. Only highly technical individuals with SQL skills and knowledge of the underlying database structures can get the answers.
• Difficulty with Upgrades: Typically there are many components in the system, supplied by different vendors. The customer becomes the system integrator. Preparing all components for a coordinated update to the system is a major IT project in itself, for every system.
• Size vs. Performance: The bigger they are, the poorer they perform. Enterprise systems today that try to provide for the whole organization, now need new systems, like in-memory dashboards and analytics engines, in order to satisfy customer demands for ‘reasonable’ performance
These problems have technical root causes in the underlying technology:
• Structural Issues in table-based data storage systems lead to enormous resource consumption in the design, construction and maintenance of tables and class models
• Namespace Issues lead to severe challenges in developing, adapting and evolving applications
• Data Integration Issues across disparate data sources require massive mapping, transformation, validation, and verifications efforts
• Query and Reporting Issues–special unique queries are needed for every single report from every single table or table sets in every single database
• Data Segmentation Issues–every table, in its normal form, by definition is a subset that must be ‘joined’ with other subsets, in order to get answers for questions that aren’t limited to any particular subset
Table-based systems enforce fragmentation, segregation and separation in our information systems. They are anti-integration, anti-connection. Every table is a silo. Every cell is an atom of data with no awareness of its contexts, or how it fits in to anything beyond its cell. It can be located by external intelligence but on its own it’s a “dumb” participant in the system–the ultimate disconnected micro-fragment accessible only by knowing the column and the record it exists in.
The alternative is to replace the data elements with information at the atomic level of the system. Instead of a data atom in a table, we have an information atom with no table. Information atoms exist in a multi- D vector space unbounded by data structures and know their context, such as a “customer” or a “product,” just like atoms in the physical world “know” they are nitrogen or hydrogen items and behave accordingly. Information atoms also know when they were created, when they were last modified, and what other information atoms of other types are associated with them. They know their parents, their siblings, and their workplace associates. They are powerful little entities and most certainly NOT fragments. Nor are they triple statements requiring endless extraneous indexing.
This insight to storing information as information all the way down–is the basic characteristic of the AtomicDB technology that can help solve the global IT problems discussed earlier and include:
• Dramatically reduce your Big Data storage requirement moving forward
• Instant access to relevant data (only 4 reads ever no matter DB size)
• Real time reports always up to date 24/7
• Create any model from any combination of portions of datasets on the fly
• Entire SQL query language replaced by one universal “Get” function
• Increase your time to market by over 50 percent
• Eliminate Security Concerns (totally new paradigm in security)
• Consolidate you disparate Databases easily (and allow interactivity)
• Aggregate your Data Accurately (being single instance removes error)
• Reduce support costs (simplicity of deployment and use)
• Easily cleanse your data (again an advantage of single instance)
• Replicate to over 1 billion devices (simultaneously near real time)