Performance issue – Custom Queryable Datsource – Unwanted Solr calls

Hello everyone,

Today I would like to share a weird issue that we faced in our project. Before explaining the issue, I would like to explain some context on how this issue was encountered.

According to the requirements, we were forced to use queries in our datasources of some component renderings. But as we all know, Sitecore does not support queries in Datasource field for the renderings out of the box. So we followed a blog which implements Queryable datasource in Sitecore MVC.

Queryable data source in Sitecore MVC

After implementing the above code logic, our requirements were fulfilled, but for some days we noticed that there were some unwanted Solr calls getting logged in the Sitecore logs and also the Solr logs. But we did not consider this as a huge issue until in the higher environments, where we had couple of CD servers were hosted in Europe and Asia regions, whereas the Solr was hosted in US region. In these environments some latency was occurring because of these unwanted Solr calls which caused the CD servers to respond slowly and also increasing the memory utilization.

This might not be exact reason for the CD server slowness response, but this could be one of the factors too. Then we started thinking to solve the same.

This was the entry in logs that was unnecessary calling the Solr :

 2019-03-26T15:02:23 PID[5736] Error 7000 15:02:23 ERROR Solr Error : [“undefined field query”] – Query attempted: [query:(.\/ancestor\-or\-self)]

And the query that we had used in the Datsource field of our Header and Footer renderings was :

query:./ancestor-or-self::*[@@templateid='{F0594847-1770-4B9E-90E5-3B3F0826EB3D}’]/*[@@templateid='{75BE5EEF-75C2-4E9C-91A6-FCEE68745E2F}’]/*[@@templateid='{0B3DAF4C-D8BC-4AE5-A4F6-98E497CC376D}’]/*[@@templateid='{872712AC-BD4E-421A-9238-DEC17090DA07}’]

After checking dumps , now we could see Footer and Header views are slow and that affects almost every page request and those were utilizing pretty heavy Sitecore queries to get the datasource for the rendering.

As the next step, we created a patch processor that uses the existing “GetDataSourceItemByQuery” in Sitecore.ContentSearch.Client.Pipelines.ParseDataSource and modified it according to our requirements. The code is same as the original except we have added a condition for datasource contains “query:” then return.

The modification was :

if (args.DataSource.StartsWith(“query:”))
{
return;
}

Here is the full code snippet :

public class GetDataSourceItemByQuery
{
public void Process(ParseDataSourceArgs args)
{
if (args.DataSource.StartsWith("query:"))
{
return;
}
List<SearchStringModel> list = SearchStringModel.ParseDatasourceString(args.DataSource).ToList<SearchStringModel>();
if (list.Count == 0)
return;
using (IProviderSearchContext searchContext = this.CreateSearchContext(args))
{
foreach (SearchResultItem searchResultItem in (IEnumerable<SearchResultItem>)this.GetItems(list, searchContext))
{
Item obj = searchResultItem?.GetItem();
if (obj != null)
args.Items.Add(obj);
}
args.AbortPipeline();
}
}
protected virtual IQueryable<SearchResultItem> GetItems(List<SearchStringModel> items, IProviderSearchContext searchContext)
{
return LinqHelper.CreateQuery<SearchResultItem>(searchContext, (IEnumerable<SearchStringModel>)items);
}
protected virtual IProviderSearchContext CreateSearchContext(ParseDataSourceArgs args)
{
Assert.ArgumentNotNull((object)args, nameof(args));
return ContentSearchManager.GetIndex((IIndexable)(SitecoreIndexableItem)Context.Item).CreateSearchContext(SearchSecurityOptions.Default);
}
<parseDataSource>
<processor patch:instead="*[@type='Sitecore.ContentSearch.Client.Pipelines.ParseDataSource.GetDataSourceItemByQuery, Sitecore.ContentSearch.Client']" type="Myproject.Foundation.Extensions.Pipelines.ParseDataSource.GetDataSourceItemByQuery, Myproject.Foundation.Extensions" />
</parseDataSource>
view raw Patchconfig.config hosted with ❤ by GitHub

After this, we asked Sitecore Support to review this implementation, but they refused to review as Custom code review was out of their scope, but they suggested the implementation will work as expected and by default that will avoid query to Solr and use the database instead. But still as it might continuously call the database, we then cached the custom datasource queryable processor also which again improved the performance.

Finally the unwanted Solr calls got reduced and the latency issue improved a bit. The main goal for us to solve this issue was, already latency issue is expected, when we have servers that tries to connect from two different locations, but on top of this, why do we want to overload it with unnecessary custom calls. So we tried to solve the issue and were successful.

Thanks to Kiran Patil for his assistance in solving the same.

This might not be an exact implementation but for our implementation this was one of the solution which improved our performance of CD servers. Please feel free to contact me if you face any issues or sharing other similar implementations.

Thank you everyone. Happy Sitecoring 🙂

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s