SimpleDB alternatives

Hi,

We are trying to add some Integration and Unit testing to our application that uses LightSpeed and SimpleDb.  However, because of the 'Eventual Consistency' it is next to impossible to write integration tests against SimpleDB.

I do not want to do complete in memory Mocking, this is Ok for Unit tests, but we have had some issues that are particular to SimpleDB that I am wanting to test against.

So, all of that... here is my question:

Does LightSpeed support M/DB ?

http://gradvs1.mgateway.com/main/index.html?path=mdb

I am considering using this as a local DB that can be used for testing and developing against.

Thanks,

-Joe Freeman

We don't currently support MDB but Rob has drawn it to our attention and we really ought to get around to it, as it would help our testing as well!  I'll take a look and post here when I have more info for you.

 I am currently trying to do some mod urlrewrite stuff in apache for MDB to make it work against the root.

However, it would be much easier if I could modify the Amazon SimpleDb Uri through the connection string parameter...

for example from my Test project:

     private static readonly LightSpeedContext<DealsModelUnitOfWork> LightSpeedContext = new LightSpeedContext<DealsModelUnitOfWork>
        {
        
            ConnectionString = @"Uri=http://192.168.1.138/mdb/request.mgwsi Access Key=XXXXXX;Secret Access Key=XXXX",
            DataProvider = DataProvider.AmazonSimpleDB,
            PluralizeTableNames = false,
            IdentityMethod = IdentityMethod.IdentityColumn,
            SearchEngineFileLocation = Path.Combine(Environment.CurrentDirectory, SearchDataDirectoryName),
            Logger = new ConsoleLogger()
        };
 

This would replace the Amazon Uri with my custom one.

Thoughts?

-Joe

Yep, that's exactly what I'm planning to do, except that I'm having trouble getting a test instance of M/DB to initialise, so I can't test it yet.

 Did you get VMWare? I had a bit of trouble, but finally got it working in VMWare server.  I had to put the directory in:

C:\Virtual Machines\mdb_22

for VMWare to find it

No, the VM stuff seems happy enough, but I made the foolish mistake of spelling "Administrator" correctly in the .conf file *grin*.  It seems to be running now -- now to see if I can get LightSpeed talking to it!

Hi Joe,

I've made some enhancements to allow it to be used with M/DB, but I need to warn you that there are some issues (which appear to be at the M/DB level) which may rule this out for you as a short-term solution.  In particular:

* M/DB does not support escaping of delimiters.  When you write e.g. where person.LastName = "O'Connor", we escape the apostrophe, so the query to SimpleDB is ['LastName' = 'O\'Connor'].  It seems that M/DB does not accept the backslash escape, and rejects this as invalid query syntax.  I haven't been able to find an alternative escape syntax that works with M/DB.  Therefore, at present, you cannot perform queries against M/DB that contain single quote characters.

* There seems to be a bug in M/DB's implementation of the starts-with operator.  It works okay if only one item starts with the specified string, but if more than one item starts with that string it appears to return no results.  Unfortunately this is fatal for sorted and paged queries: because SimpleDB requires us to use the sort attribute in the query expression, we add a ['Xxx' starts-with ''] clause to every query that sorts on Xxx.  This of course matches multiple rows, and therefore causes M/DB to return an empty result set.

I may be missing something here and we would of course be happy to work with you and the M/Gateway folks to resolve these problems (and if we are doing something wrong then of course we will fix it), but right now I guess these issues will significantly limit how much testing you can do using M/DB.

Anyway, if you still want to give it a go using simpler queries, here is what you need to do:

* Get a nightly dated 16 May 2009 or above.  These should be available from about 1430 GMT.

* In your connection string, add "Url=xxx;Signature Version=1" (where xxx is your M/DB URL, e.g. http://192.168.xxx.xxx/mdb/request.mgwsi).  Note you must specify the Signature Version as M/DB does not appear to support the default version (2).  Also, note the key is Url with an L, not Uri with an I.

Please let us know of any issues you run into; however because of M/DB limitations such as the ones noted above we may not be able to resolve issues related to queries.  You can capture the generated queries using LightSpeedContext.Logger and I would suggest that if a query runs okay against SimpleDB but fails or runs incorrectly against M/DB then you should raise the issue initially with the M/DB folks (we will of course be happy to help if necessary).

We have not yet updated SimpleDB Management Tools to work with M/DB, but will look at that in the near future.

Hi Joe,

We have now also updated SimpleDB Management Tools so if you run into query problems you can use that to test the query against both SimpleDB and M/DB to see if it's a LightSpeed or a M/DB issue.

 Thanks Ivan,

 I will try this all out tomorrow. 

 I believe this will still work for us, because:

1) we are only using M/DB as a testing platform

2) we will have control over all the test data, so the single-quote thing can be worked around for now

3) we do not currently use the 'starts-with' feature anywhere in our application

If I run into any  issues that cause errors, I will try to figure out if it is a M/db or a lightspeed by using the Management Tools.

Thanks at lot!

-Joe Freeman

Hello Joe,

We've been talking to Rob at M/Gateway and he's fixed the key issues that were affecting using M/DB from LightSpeed:

1. Single quotes in query arguments now work

2. starts-with now works which means sorting should be okay

3. You no longer need to override the signature version or method -- the defaults just work

These fixes are in build 26 of M/DB, available from http://gradvs1.mgateway.com/main/index.html?path=mdb (Download tab).  You shouldn't need to update LightSpeed or SimpleDB Management Tools.

One outstanding issue is that M/DB still does not appear to support paging using MaxNumberOfItems and NextToken.  We're working with Rob to determine whether this is an M/DB issue or whether we are doing something wrong, but apparently there are some gnarly technical issues and Amazon's responses on this area are a bit confusing.

We'd like to thank Rob and M/Gateway for the great support and rapid turnaround on this.