When dealing with large volume of inter-connected data, there is a need for a better mechanism to query such data via APIs. Thankfully, GraphQL offers a solution. With a graph database that supports native GraphQL Data Ingestion, it is possible to build a data source from multiple enterprise applications for performing real-time business process monitoring.
More...
This post was originally published in DGraph
Use Case: GraphQL Data Ingestion
Problem Statement
Big data is increasingly showing the characteristics of connected data, organized as graphs, resembling bits of knowledge, linked as a network of entities and their relationship. This arrangement makes it harder to gain insights from large quantities of inter-connected data.
Realization Approach
A graph database like Dgraph, which is built for the cloud, and offers native GraphQL support to enable data ingestion based on simple and standards-compliant GraphQL queries, deployed on a massively distributed architecture.
Solution Space
A readily available GraphQL API interface for ingesting very large volume of graph data, with support for flexible schema and intelligent resolvers implemented by following graph relationships.
Modern business processes often span multiple enterprise applications that are deployed either on-premises or in the cloud, with each application focusing on a specific domain such as order management, shipping, or payments. Middleware components are used to coordinate between the applications to synchronize the overall business process. Customers access the process through various channels like mobile apps and Facebook Messenger.
However, real-time monitoring of these distributed asynchronous processes can be challenging. Late visibility into the process status can lead to poor customer experiences and unnecessary interference from call center agents. A GraphQL-based solution using Dgraph can help provide timely visibility into the distributed business process.
This example demonstrates how to monitor distributed business processes across multiple enterprise applications using Dgraph capabilities such as the GraphQL API and Lambdas. You will learn how to leverage these components to store, inspect, and create incidents for effective process monitoring.
Note: Dgraph Cloud provides an easy way to create the schema and run queries and mutations. You can follow Quickstart steps to get started.
Process Monitoring in the Modern World
Process Monitoring Challenges
A simple approach to creating a process monitoring solution would be to route all your important logs, events, and message payloads from respective applications into a common storage layer and then use this layer to drive queries.
However, there are several challenges involved in this process.
* Data from different apps is in inconsistent formats (fixed-width text, XML, JSON, etc.)
* Data fields are not standardized (customer name could be in different fields)
* Values are not standardized (currency code could be 3-digit code or text)
* IDs are not consistent across apps (order number would differ between apps)
These inconsistencies make it difficult to aggregate data and get a holistic view of a distributed business process. A GraphQL-based solution would need to account for the varied data formats, fields, values, and IDs in order to combine information from multiple enterprise applications.
In order to build an effective monitoring solution that addresses the challenges mentioned above, you will need to involve team members from these distributed apps and find a way to incorporate their knowledge about structure, content, interpretations, etc. in your solution.
You could consider the methodology explained here to use a GraphQL API running on Dgraph Cloud GraphQL for a coordinated solution to this problem.
Process Monitoring using Dgraph as GraphQL Data Ingestion Layer
You can use Dgraph Cloud and GraphQL to build a robust process monitoring system. The solution will have two main parts:
- A GraphQL API for ingesting messages. The API will accept messages from different apps in various formats. GraphQL will allow you to define interfaces tailored to each app's data. For unstructured data like logs, the interface will be simple strings. For structured data like incidents, the interface will match the data format.
- A message processing workflow. The workflow will analyze message content using Dgraph Cloud. It will consist of small, discrete steps implemented as Lambdas on Dgraph Cloud. The Lambdas will extract structured data, handle errors, enrich data by accessing external sources, transform data, and store the results.
Overall, the GraphQL API will provide a standard way to input data, then the message processing workflow will handle the data consistently and store the results. This will enable monitoring of the distributed business process.
The Dgraph Cloud-based message processing workflow will produce a graph that will provide near real-time and correlated visibility over your entire enterprise. You can use this graph to drive incident creation (ticketing) as well as extract trends for analytics.
A less understood aspect is that many modern apps and middleware have sophisticated operating capabilities. For example, modern cloud components such as queuing engines are capable of automatically managing issues if the right levers of throttling or scaling are engaged.
This implies that modern business process monitoring solutions need to treat these levers as first-class digital citizens. You can choose to connect levers such as throttling, scaling, etc. available in the enterprise directly to this graph. This creates a feedback loop where-in the graph can be used to measure and tune the effects of engaging these control levers. For example, you can answer questions like, “Did the throttling applied on the middleware help in reducing the end-to-end processing time on the backend DB?”.
Finally, you can use the Dgraph Cloud GraphQL API to drive these visibility and feedback-related processes. You do not need to take any overhead for provisioning an API for your monitoring teams as Dgraph Cloud will be providing a GraphQL API for you out of the box.
You can read the original solution brief to understand the GraphQL schema details and step-by-step walkthrough of the code.
