How to report on Contextual Search usage

April 19, 2024

Do you use Contextual Search or Agent Assist features? If yes, maybe you wondered how you find out if it is used and how much it helps. There is a page on ServiceNow documentation site about how to report on Contextual Search but it was not detailed enough for me. Also, I could not find any dashboard about Contextual Search baseline. In this blog, I am mostly using Contextual Search, but the same applies for Agent Assist (used for example in Service Operations Workspace).

Therefore, I did dig into the tables and data, tested the behavior so that I know what kind of answers we can deliver with reports on that data.

Is Contextual Search helping to deflect incidents?

Is the content offered by it actually marked helpful by users?

What are the most helpful articles or the most used catalog items?

Before we start building any reports, we need to understand the table structure and what data we have available. Below, I created a simple entity relationship diagram (hand-written - yes, sometimes I still like good old paper) that shows the main 2 tables and key attributes that we can use.

Relevant document [cxs_relevant_doc] table stores one record for one contextual search usage by a user in a record, like Incident or in Record Producer form. This record then might have multiple "Relevant document details" linked to it explaining what the user did in that Contextual Search usage, like previewed article, attached it, or marked it as “This helped”.

As explained above, Relevant document detail [cxs_rel_doc_detail] is the key table then to report on user activities within the Contextual Search and Agent Assist.

What are the most interesting attributes of Relevant Document table you want to report on?

• “Displayed on” - find out where users use Contextual Search the most - Is it Incident form in platform UI? Is it within workspace, or in a record producer?
  • Tip: To make the reporting easier, we would need to separate the “sys_id” of the record producer from this field’s value (e.g., record_producer:[sys_id]) so that we can also report on if it is “incident” record producer, or at least show its name in the report instead of “noname” identifier.
• "User” - find out how many unique users used Contextual Search and where. Are those itil users, or end users?
• “Relevant to” & “Source doc table” - this one is interesting, because it seems it stays empty when there was actual incident deflection! It means, if user was on a new incident form, or record producer form, but at the end did not submit the incident creation, this stays empty. Did you want to measure Incident Deflection rate? Take it! If the incident got submitted, this will contain the link to it.

What are the most interesting attributes of Relevant Document Detail table you want to report on?

• “Relevancy type” - answer questions like how many times users clicked attach, preview, this helped or order buttons. Pretty cool for showing the most helpful articles or catalog items.
  • Attention: “This helped” in Contextual Search is NOT same as “Mark helpful” on Knowledge Articles. “This helped” does not actually create “Knowledge Feedback”. It only tracks it here in this contextual search table.
• “Relevant document” & “Relevant doc table” - this is a bit confusing naming, as it duplicates the name of above table. In fact, it means something else here. It stores the related Knowledge Article or Catalog Item that was previewed, ordered, attached or marked as helpful.
• “Search term” & “Original search terms” - do you want to know if users modified the search query? Thanks to these two attributes you can. If they differ, user obviously changed the query in Contextual Search.
• “Position” - stores the position of the item. This can be helpful to evaluate search relevancy. Documentation says that “The information in the table can help you find the instances where users ran a search but did not take any action. In those cases, the value of the Position column is -1.”. However, based on my testing it is not fully correct. Value “-1” is there for “search” actions when I modified the search term, and it was there even when I did action within that search - however that was tracked as next “Relevant doc detail” record. So to answer question like “how many times users did not took any action”, you would need to actually report on “Relevant Document” records that have only one “Relevant Document Detail” record of “search” relevancy type.

So what would be my recommended reports to built and questions to answer?

• Where are the users using the Contextual Search?
• How many unique users did use it?
• What percentage of incident creations via record producers are deflected?
• What percentage of contextual search is helpful or results in attaching or ordering?
• What knowledge articles are the most helpful in contextual search?
• What catalog items are the most helpful in contextual search?

Here is an example of my testing Dashboard that I built on my PDI when working on this article. Not really a dashboard that I would offer to a customer, but at least it gives you a visual idea of where to start.