Custom Search Engine with Paging and SimpleDB

We are creating a custom search engine for Lightspeed following this example:

 http://www.mindscape.co.nz/blog/index.php/2009/02/25/lightspeed-writing-a-custom-search-engine/

All works well, untill we try to add paging. 

I believe 2 things are needed in order to support paging in a custom search provider:

1) add a method to the SearchEngineBroker Class:

public IList<SearchResult> Search(string query, int offSet, int limit, params Type[] types);

2) Add 2 properties to Page Class

public int GetLimit;

public int GetOffset;

However, they might have to be added to the Query class due to an error I am getting "This query is not supported on non-SQL data stores " because we are using SimpleDB:

Here is a sample of the code that we want to be able to execute against a custom search provider:

   List<MyData> myList = new List<MyData>();
   Query query = new Query
   {
    IdentifiersOnly = true,
    SearchQuery = q,
    Page = Page.At(page,count)
   };
   
     var results = Data.Search(query, typeof(MyData));
     foreach (var result in results)
     {
      myList.Add(GetMyData(new Guid(result.EntityId)));
     }
   return myList;

-------------------------------------------------------------------------------------

 and I want to do something like this in the Search Engine:

    var hits = (queryFilter != null)
          ? searcher.Search(queryParser, queryFilter)
          : searcher.Search(queryParser);

    if(count > hits.Length()) count = hits.Length();

    results = new SearchResult[count];

    for (var i = offSet; i < count; i++)
    {
     results = new SearchResult(
       hits.Doc(i).GetField(KeyFieldName).StringValue(),
       hits.Doc(i).GetField(ScopeFieldName).StringValue(),
       hits.Doc(i).GetField(EntityIdFieldName).StringValue(),
       hits.Score(i));
    } 

-----------------------------------------------------------

Thoughts on this? Suggestions?

Thanks,

-Joe Freeman

Hi Joe,

Thanks for the long post explaining what you're trying to achieve :-)

I'll just take a moment to explain how things should operate with the search engine. The seach engine is designed to provide full text capabilities over the entities in your data store (be that the full text is stored in Lucene, SQL Server Full Text index, some random other store).

The reason paging is not supplied on the search engine object itself is because the search gets refined by the query you're executing against the database. Take for example this query (pseudo code):

Query q = new Query(){SearchText="joe", PageSize=10, Page=2,  QueryExpression="Age is larger than 15"}

In this example, the search index will simply return all the Id's that match the query for "joe". This could be 300 integer values for example. LightSpeed will then automatically perform the query that states that all entities within that set of values should have a property of "Age" that is greater than 15.

The issue here is if the search engine itself was to limit based on paging, for example to 10 items at page 2, and then only 3 of those results actually matched the query expression then you would say your page size is 10 but only have 3 results.

The query you're providing should work fine for providing paging. I have spoken with Ivan who implemented the SimpleDB provider and he believes this issue you're seeing with paging may be because you're using an older version of LightSpeed. Could you try grabbing the latest nightly build?

The initial design decisions around the incorporated search engine was to provide a basic but extensible full text search that could run on any of the databases that we supported. We're always happy to hear suggestions around how we could extend the search engine capabilities to provide more for our users but I suspect the issue of paging on the search provider is misplaced.

Having said all this, I may look at the following change which may help you:

• Currently if you search with identifiers only and provide no other constraints (ordering, paging, query expressions, etc) then we do not attempt to hydrate entities based on the result set
• We could change this to suggest that if paging is provided that we do this in memory against the identities returned (note, the results from the search provider would need to be managed by you to ensure this worked reliably).
  

What do you think? Initially, it should just work with what you're trying to do with the latest version of LightSpeed. Secondly, we could look at making the enhancement suggested here if it would help you further? I just want to ensure we avoid duplication of paging code in the core querying engine and then again in the search engine with it only applying in each place given a certain set of circumstances - it makes the code harder for new users to pick up and understand.

Appreciate your feedback on all of this and I may be on the complete wrong track! If you still have issues with the latest nightly build could you please attach a small repro case so we can get to work on ensuring it works for you?

Kind regards,

John-Daniel Trask

 Hi John,

Thanks for your well written response.  We are working around this issue by calling the custom search engine directly:

----------------------------------------------------------------

            IList<Object> objects = new List<Object>();

            IPagedSearchEngine se = Unity.Container.Resolve<IPagedSearchEngine>();
            var results = se.Search(q, page, count, typeof(Object).FullName);
            foreach (var result in results)
            {
                try{objects.Add(Data.Questions.Single(x => x.Id == new Guid(result.EntityId)));}
                catch (System.Reflection.TargetInvocationException){}
                catch {throw;};
            }

            return objects;

----------------------------------------------------------------