Lightspeed Full Text

Hi

I am implementing a full-text search using LightSpeed 4.0.1725.22176.

The problem is that the lightspeed query returns 0 results when it should be returning some. There is a large SQL filter in place that restricts the results from document pool of around 800,000 to 1800 or so.

It appears however, that Lightspeed does the full-text search first (which returns around 22K documents). However, it only passes the first 256 ids to the SQL filter, which then filters them all out (publication date, document channel etc). At that point, the query ends.

I am doing this with Query.

How do I get the proper behaviour here?

Cheers Andy

So I found out the cause of my problem (and now have a new problem). Because I had to write my own Lucene search provider that uses the new 3.x API, I was using a TopDocCollector (which had an unfortunate hard-coded value of 250 docs), it was only reporting 250 documents to the LightSpeed query engine.

Now that I have changed it to report all 22K documents, Lightspeed now sends all 22K Ids in the query as named parameters. Sql Server only allows 2100 parameters, so this presents a problem.

How does Lightspeed normally handle full-text searches that return thousands of documents?

I presume you are not applying any paging? We use the paging to restrict the id's being sent down to the database for the In(Ids) part of the query. So thats certainly the primary way you would handle this.

If paging is not appropriate because you actually want to deal with all of the entities then you will need to implement a custom approach to fetching the entities back in a paged manner and combining them yourself. You can do this by making a call yourself to context.SearchEngine.Search(searchQuery, entityTypes) and then you will have a list of resulting entity type/id pairs which you can use to filter on.

Can you elaborate on what you are looking to achieve? Would paging be appropriate for you?

Jeremy

I have uploaded a repro of what I am trying to accomplish.

Basically, the Lucene query returns a large number of documents (>2000) that I then ask SQL to filter further (In the repro I use a simple on/off boolean).

Essentially, a Lightspeed query that says "Of all documents with the keyword K that match SQL criteria X, please return rows n..n+100"

Thanks that helps clarify things.

You are going to need to handle this yourself because we cant limit the items when there is either an ordering applied in the query or there is a database criteria that needs to be applied. In those cases we have to send down all items found from the search index which is going to be limited by the number of parameters. For the example in the test you could break this into batches and then union it client side but this will vary depending on what else you need to do in your actual system.

e.g.

[Test]
public void GetCountWithFullText()
{
  using ( var uow = _context.CreateUnitOfWork() )
  {
    var results = _context.SearchEngine.Search("quick", typeof(Item));
    long count = 0;
    int processed = 0;

    do
    {
      var query = new Query(typeof(Item), Entity.Attribute("Id").In(
        results.Skip(processed).Take(1000).Select(r => r.EntityId).ToArray()) && Entity.Attribute("Show") == true);

      count += uow.Count<Item>(query);
      processed += 1000;
    }
    while (processed < results.Count);
  }
}

Jeremy

Hi Jeremy

Is there a way I can add a Table Valued Parameter to the Query object instead of an array? Passing that many parameters seems hideous, even in batches of 1K (Which has its own impact on paging).

You can use TVP's in conjunction with stored procs (e.g. http://www.mindscapehq.com/forums/thread/479220) but not as part of a normal query the TVP would need to map to a known table type and we dont have a way of carrying through the type name to set it on the ADO.NET parameter.

Jeremy

Unfortunately I can't use a stored procedure as the where clause is determined by the search options selected by a user.

I think I may have to drop the LightSpeed full-text integration unfortunately! It looks like I am going to have to put a lot more information into the Lucene index so I can help filter the candidate documents in the same way SQL will.

Boo! Oh well. Its' not like EF would do this any better.