Monday, August 23, 2021

Troubleshoot Azure Cognitive Search Latency Issue

General Performance Issues:


If a service is not under consistent traffic, query latency will appear to be much slower than desired.  This is because the service is cold, and caches must be loaded to serve up their response.  The service is optimized for performance under load.


If the query sometimes is slow:


Scenario1: When there is a spike in request volume at the search service level, either indexing or query, it is expected that latency can also increase.  In many cases, the recommendation here would be to accept a delay on updates to make more room for search queries, since an actual end-user is usually waiting for a search result and indexing operations are typically asynchronous.

We guarantee at least 99.9% availability for index query requests when an Azure Cognitive Search Service Instance is configured with two or more replicas, and index update requests when an Azure Cognitive Search Service Instance is configured with three or more replicas. No SLA is provided for the Free tier. In short, 2 replicas to read and 3 replicas to read/write. For more info, please refer to SLA for Cognitive Search | Azure


Scenario2: When the number of requests is too high for the current configuration of the service, additional requests may be throttled. Queries become slow for a short period of time around the time the service was throttling requests.  Suggest assessing the acceptable percentage of throttled requests and add replicas to handle more traffic once that acceptable threshold is exceeded.

To get a better understanding of Search service capacities, please refer to Estimate capacity for query and index workloads - Azure Cognitive Search | Microsoft Docs.





If the query is slow at the first time:


Scenario1: Make sure no cache mechanism is triggered, especially the query file from the frontend(browser)instead of the backend server.


Scenario2: Pay attention to the client environment. If client such as Azure app service or Azure VM which integrate with VNET, and there are custom DNS servers configured on that VNET. The DNS server is not working properly, it will also cause a delay when the first-time query to the Search server.





If the query is always slow:


Scenario1: Query response size is too big. Suggest reducing the overall size of the response will also reduce the amount of time it takes to send the response over the network.  Some tactics for reducing response size:

  • Reduce the number of fields retrieved.  Especially if a field is not displayed in the search UI.
  • For large string fields, only retrieve hit highlights and not the full contents of the field.  It may not be needed for the UI.
  • Keep the number of documents retrieved as small as possible.  Default is 5 but avoid going much higher than that.  

Scenario2: The filter criteria for each query are too complex, we should simplify the filter criteria for each query. 


Scenario3: The number of searchable fields that are enabled on the index or targeted for a search query.  Reduce it can improve performance particularly at a high volume of data in the index, by reducing the amount of content in the index at search time. We can check it via the portal:




Some recommendations:


1. Flatten your schema as much as possible.  You should not expect to keep your schema the same as your underlying relational database, for example. The search index schema should be optimized for performance for the specific search queries you plan to run for the search experience of your application.

Such as the schema in your SQL, when creating search index, some of the fields is not necessarily to add to return.



2. Pre-process your data on the way into your index wherever possible.  For example, if you are looking to find names in a very large string field, you could extract the names at indexing time into a string collection and search in that collection for the name later.

Skillset export a new column will help filter source


3. Add a partition or move to a higher service tier.  You may be getting all the performance you can out of your service and configured and may want to consider scaling up.  Adding a partition will improve query performance slightly since your data is now divided among more search units and adds more computing power to your service.  Each search unit in higher tiers has more processing power and, as a result, individual queries may consistently perform better.

Posted at