M/DB: MaxNumberOfDomains=1 ?

Hi,

I'm using the node-mdb virtual appliance from M/Gateway (as described here) to run a local instance of what will later be an AWS SimpleDB database.
Now when I point the SimpleDB Management tools to it, everything works as long as there is exactly one domain. Otherwise I get a timeout. (I also have timeout issues with LightSpeed. I stronly suspect that this is due to the same reason.)
I think I see the problem. When I look at the trace from the node-mdb terminal window, there is the argument 'MaxNumberOfDomains=1' present on the request:

Is this something on the Mindscape side of things or has it to do with node-mdb? Because I'm not sure about this, I posted this problem also over at the M/DB Community Forum.

I'm using these versions:
SimpleDB Management tools: v1.0.550.18139, VS2010
LightSpeed: v4 pro
AWS SDK: 1.3.10
M/DB virtual appliance: latest version from here, not modified

Thanks in advance

Thomas

Ups, there was a screenshot pasted into the previous post. Seems that this doesn't work that way. But I think the problem should be clear anyway...

Hi Thomas,

The only time we set MaxNumberOfDomains is in a connectivity testing method (where we use it as a 'dummy' request and just throw away the response).  When we subsequently enumerate domains, we don't set MaxNumberOfDomains -- we will take as many as the server returns.

Basically what you should see is a ListDomainsRequest with MaxNumberOfDomains=1 as Visual Studio 'connects,' followed by a ListDomainsRequest with no MaxNumberOfDomains when you open the Domains folder.

Can you tell if M/DB is sending a response to that initial ListDomainsRequest, or is it hanging without sending a response?  Or does it send a response to the initial 'max 1' request and then hang on the subsequent 'unbounded' request?

Regarding LightSpeed, do you get the timeouts when using the designer (in which case it almost certainly is the same issue), or do you mean you get timeouts when issuing queries at run-time?  If the latter, then something more fundamental is going on, as LightSpeed doesn't send a ListDomainsRequest at runtime -- it just sends a SelectRequest.

Hi Ivan,

thx for the quick response.

I see the 'MaxNumberOfDomains=1' argument also when I want to create a domain on the connection via Mgmt. tools. The picture is this: Creating the first domain works seemlessly. When I try to create the second one, I get a timeout. The last line that appears on the node-mdb terminal in that case is: 'Action=ListDomains' (the 'MaxNumberOfDomains=1' argument is present in the request). It looks as if node-mdb does actually create the new domain, but doesn't send a response thereafter. Because when I delete the first domain via Mgmt. tools, I can immediately see the second one...

Regarding Lightspeed: The timeout appears at run-time. If my code wants to create some datasets in a loop (there is only one domain), then the first gets created, but then I get a timeout. Seems that in this case also there is no response from the node-mdb instance for some reason (I can see that from the testlog). Would it make sense to open up an extra thread for that over at the LightSpeed forum?

Regards

Thomas

Hi Thomas,

This is looking to me like a bug in M/DB.  If you issue a ListDomainsRequest with the MaxNumberOfDomains smaller than the number of domains, then it appears not to send a response. I have tested this using a console application, with no Mindscape or Visual Studio code, and I still see the timeout, so the issue is either within M/DB or due to some incompatibility between M/DB and the .NET AWS SDK library -- I haven't put a network analyser on it but since I don't see a return trace in the M/DB Appliance I'm guessing it's the former.

(The hang when creating a domain, by the way, is because the connectivity check gets performed as part of the refresh after creating the new domain.)

I can work around this by removing the MaxNumberOfDomains limit in the connectivity check, or upping the limit beyond what you are using, though this might have a small performance implication for other users.  If I set the MaxNumberOfDomains to 10, would that be enough for you, at least as an interim measure?  If I know Rob he will have the issue fixed in about five minutes anyway!

I'll post an update to the M/DB forum thread with what I've found.

(I am still looking into the LightSpeed runtime issue -- will let you know!)

Regarding the LightSpeed runtime issue:

I haven't been able to reproduce the timeout.  Could you provide a short console application that exhibits the problem, including instructions on what domains we need to create?  You can attach a zip file via the Options tab (please remove all binaries first).  If you could also include your M/DB trace log then that would be handy too.

However, there does seem to be some another problem.  M/DB seems very sensitive to whitespace in Select queries.  It looks like M/DB requires exactly one space between terms, and treats newlines as nonexistent rather than as spaces.  For example "SELECT * FROM Game" works fine but "SELECT *\r\nFROM Game" produces a query syntax error and "SELECT *  FROM Game" (with two spaces between * and FROM) produces 'domain does not exist.'  Our query formatting isn't quite regular enough to satisfy this (we don't collapse multiple spaces, and we treat newlines as whitespace).  I suggest this would be best addressed in M/DB, but if Rob doesn't have the time, or if he feels the Amazon 'spec' (hah!) requires this careful formatting, then we can try to work around it at our end.

Do you want to flag this up on the M/DB forum?  I've attached my repro app in case it helps.

Hi Ivan,

thanks for finding the problem.

No need to hurry with a workaround/hack on the Management Tools side - I won't be able to check this before Thursday evening anyway, so I'll rely on Rob for that...

Hi Ivan,

regarding the LightSpeed runtime issue:

No problem, I will do a little repro app for you. Because my time is rather limited at the moment, this may take some days - presumably 'til next weekend.

In the meantime: could you tell me (or point me to something helpful) how I can get the M/DB trace log from the Ubuntu VM into Windows. I'm a Windows guy, and therefore I'm somewhat helpless in Linux when there is more required than starting a pre-configured VM and typing some pre-defined commands...

Uh, sorry, I'm a Windows guy too.  Rob may be able to give you some advice on capturing the trace log and transferring it from the VM to Windows.  I can't even figure out how to resize the VMware window *grin*.

Hi,

attached is the solution which reproduces the aforementioned problem with LightSpeed.

I couldn't reproduce it with a simple console app, so I attached a stripped down version of my original trial solution here. The issue must somehow relate to the slightly more complicated things that I do in there, but I've no idea what this could be.

The solution requires one M/DB domain: 'User'. If you created the domain and then run the tests in 'UserFixture' (sorry, this will require the Gallio/MbUnit platform), you will see that the second test ('CreateUsers()') will hang with a timeout after having created the first entity. Looks like again there's no response from M/DB...

Thanks in advance
Thomas

Hi Thomas,

This looks like it could be a race condition within M/DB to do with the DeleteAttributes operation -- but there is also an quirk on the LightSpeed which is triggering the problem.

The timeout with your example is not to do with the loop or with multiple domains, it is to do with the fact that the users you are creating have some null fields.  (The Thomas entity, by contrast, populates all its fields.)  If you change the loop to populate all the user fields, then it runs successfully.  The problem is that LightSpeed sends a null by sending a DeleteAttributes request.  So if you look at the log, you will see a PutAttributes for all the attributes you do populate, followed by a DeleteAttributes for the attributes you don't.  Now M/DB normally handles the DeleteAttributes just fine, but it seems to have a problem when doing a DeleteAttributes on a record that has only just been created.  So I think there is a bug in M/DB here.

Again, we can demonstrate the timeout behaviour outside of LightSpeed:

    [Test]
    public void MdbPutDelete()
    {
      AmazonSimpleDBConfig config = new AmazonSimpleDBConfig
      {
        ServiceURL = "http://192.168.13.107:8081/"
      };
      AmazonSimpleDBClient client = new AmazonSimpleDBClient("administrator", "mdbsecretkey", config);
 
      PutAttributesRequest put = new PutAttributesRequest
      {
        DomainName = "User",
        ItemName = "User" + Guid.NewGuid().ToString(),
        Attribute = new List<ReplaceableAttribute>
        {
          new ReplaceableAttribute { Name = "Location", Value = "Wellington" }
        }
      };
      client.PutAttributes(put);
 
      DeleteAttributesRequest delete = new DeleteAttributesRequest
      {
        DomainName = "User",
        ItemName = put.ItemName,
        Attribute = new List<Amazon.SimpleDB.Model.Attribute>
        {
          new Amazon.SimpleDB.Model.Attribute { Name = "AboutMe" }
        }
      };
      client.DeleteAttributes(delete);
    }

The PutAttributes succeeds, but the DeleteAttributes times out.

That said, the LightSpeed DeleteAttributes call should be completely redundant when creating a new entity; it's only needed when updating an existing entity and nulling out an attribute.  (And M/DB handles that case fine.)  So we should be able to enhance LightSpeed and work around this bug at the same time.  I'll take a look at this, but I'd still encourage you to flag this up to Rob and see if he can figure out what is going on.

Okay, some more information.  The reason we are sending that DeleteAttributes on an 'insert' is because of your FindById() call in MakePersistent.  You are expecting this to return null if there is no item with the specified ID (and so was I, to be honest!).  But it turns out that SimpleDB and M/DB will always return a value from a GetAttributes() call -- they do not distinguish 'item exists with no attributes' and 'item does not exist.'  So FindById always returns an entity, so MakePersistent always results in an 'update' rather than an 'insert,' so we always issue the DeleteAttributes call.

To work around this, you need to change your test for 'does the entity already exist.'  There are a couple of ways to do this.  One is to issue a Count query instead of doing a FindById; obviously, the trouble here is that if successful this will result in two queries to SimpleDB, one for the count and one for the details (though the latter will check the cache).  The other is to perform a Find query, but a simple Find by Id will still get mapped down to a GetAttributes; you would need to add a spurious query condition like Name == "Some Name That Will Never Ever Exist," which is both ugly and defeats caching.  You could also replace your natural key with a GUID key, make Email an attribute, and perform a search by Email.  This avoids natural keys which LightSpeed is never fond of, but defeats fast L2 cache lookups.

The best solution is probably for us to fix FindById so that it returns null if SimpleDB returns no attributes.  This would be legitimate because SimpleDB doesn't have the concept of 'item exists with no attributes' -- such an item is non-existent.  However it could affect compatibility (what if an existing application depends on FindById returning vacuous entities on SimpleDB?).  So I'm going to take a look at implementing an option to say 'if GetAttributes finds no attributes then return null instead of a vacuous entity'.  This would allow you to keep your existing code.  I'll keep you informed.

Okay, I have added a new connection string option to specify the handling of zero-attribute entities.  If you add:

Materialize Zero Attribute Entities=false

to your connection string, then LightSpeed will check for zero-attribute results during a FindById, and return null instead of a vacuous entity. (I realise the connection string is a bit of an odd place to put this, but ProviderOptions is for per-query overrides and you really want a global setting.)  This should allow your existing code to work correctly, which will in turn avoid the spurious 'DeleteAttributes because we think we're doing an update' behaviour, which will in turn avoid the M/DB put-and-delete bug.

We will probably make this the default in a future version of LightSpeed, but for compatibility reasons you will need to specify it explicitly for now.

The new option will be available in the next nightly build.

Hi Ivan,

Thanks for the reply and the resolution. I will check it out and give you feedback the next days. - I was always wondering why there were so many 'Delete' requests in the test log, but I didn't really care as long as everything worked just as it should...

I also will issue this on the M/DB community forum. (Rob hasn't answered yet to my previous questions (regarding the 'MaxNumberOfDomains' issue) - maybe he's in holiday or something. But that's not much of a problem for the moment.)

Regards
Thomas