Datorios, a developer of data observability and data quality technology for real-time business operations, has made new observability innovations available for the open source Apache Flink stream processing system.
Announced at Ververica’s Flink Forward 2024 underway in Berlin Germany, the new technology from Datorios equips businesses with the observability required to ensure that Flink-powered real-time AI applications drive accurate, rational business automation on a continuous basis.
Datorios correlates information about real-time system behaviour-data, code execution, infrastructure–all in one place, making it vastly simpler to pinpoint and resolve problems with operational AI applications.
Agentic AI is the next step in the evolution of AI applications. Agentic AI goes beyond generating answers to questions; it uses AI to create and execute plans to achieve specific goals. It automates decision making and operations based on those decisions. Gartner predicts that by 2028, 15% of day-to-day work decisions will be made autonomously via agentic AI.
Responsible Real-time AI Requires Apache Flink Observability
Apache Flink is a critically important part of the real-time agentic AI infrastructure. It is used to combine and shape data from across the enterprise, giving AI agents the data and understanding they need to take action. With businesses becoming increasingly dependent upon AI systems, it is essential to be able to observe the inner workings of Flink programs in order to understand what decisions agentic AI systems take.
Ronen Korman, CEO and co-founder of Datorios, said: “No responsible business would operate without cybersecurity in place today, and likewise, no responsible business should let AI drive operations without observability in place.
“Agentic AI holds tremendous positive potential for business, but you absolutely need visibility into how AI systems are automating your operations. Datorios gives you the visibility for the needed operational quality no business can leave aside.”
New Correlated Traces and Data Lineage Analytics for Apache Flink
As real-time data streams into and out of AI models such as LLMs, it is processed by Apache Flink programs. From time to time, it becomes necessary to observe how these Flink programs are behaving–in response to a system crash, or a customer complaint, or a regulatory audit request. The new observability capabilities from Datorios provide this insight in interactive dashboards purpose-built for real-time stream data processing in Flink. This enables businesses to quickly pinpoint the root cause of an issue and resolve it.
New functionality includes:
Correlated traces – the cause of an AI system failure can nearly always be attributed to a problem with the data, or the Flink code that acts on the data, or the computing infrastructure on which Flink runs. Datorios provides a unified platform that correlates data traces across data, code, and infrastructure, allowing Flink teams to detect, isolate, and resolve issues quickly. With Datorios, operational complexities are unraveled, empowering Flink users to maintain the operational continuity they need to succeed.
Lineage analytics help Flink teams trace the journey of data within Flink jobs, which makes it clear when and why data changed, and who on the team is most familiar with the offending code or data source so that the issue can be fixed as quickly as possible. Lineage analytics capabilities include:
(Photo by Murray Campbell)
Want to learn more about AI and big data from industry leaders? Check out AI & Big Data Expo taking place in Amsterdam, California, and London. The comprehensive event is co-located with other leading events including Intelligent Automation Conference, BlockX, Digital Transformation Week, and Cyber Security & Cloud Expo.