i am a developer in need of some guidance in building a custom LINQ Provider.

the backend data source for my provider is Azure Table storage. a common client-side query might look like this:

var query = (
from m in Medication.Samples
join d in Demographics.Samples on m.UID equals d.UID
where (m.Dosage == "250"smilie && (d.Age > 50)
select new { ... });

my source types above implement IQueryable. when Execute() gets called on my IQueryProvider, i am using MetaLinq to serialize my Expression Tree to the server-side (an Azure Worker Role) where the query execution actually occurs.

on the server-side, i want to essentially partition the above where clause into its source-specific components (e.g. Dosage with Medication AND Age with Demographics), each which translates to a seperate Azure Table (since Azure Tables dont support Joins).

i will then need to query each table individually using its specific components, bring these independent results back into memory on the server-side, and finally perform a join using LINQ to Objects, LINQ to XML, or LINQ to DataSets on the in-memory result collections. i then have a helper class which would translate the client query's projected anonymous type into XML for transmission back to the client-side.

my first question is do you know of a clever/elegant way to accomplish this task of breaking up the where clause into separate queries that will work against Azure Tables using System.Data.Services?

basically, i need to accomplish the following:

1) get original query Expression Tree on server-side (via MetaLinq).

2) breakup where clause into N queries, one for each Azure Table involved. the correct sub-expressions from the where clause need to be applied to each sub-query.

3) execute N sub-queries and then inject the results into memory on the server-side, where they can finally be queried together using the specified Join (via LINQ to Objects, LINQ to XML, OR LINQ to DataSets).

4) serialize the results as XML and return to the client-side.

i am fine with the current operator limitations within the where clause for Azure Tables, but Join support is a must, so i need to be able to elegantly support this capability. and i have been going a bit nuts trying to figure out a good way to achieve steps #2 and #3.

also, there is something else that i ran into. given this query:

string sPrefix = "x-";
var query = (
from m in Medication.Samples
join d in Demographics.Samples on m.UID equals d.UID
where (m.Dosage == sPrefix + "250"smilie && (d.Age > 50)
select new { ... });

when i inspect the expression tree, i see a reference to the 'sPrefix' variable, but NOT the actual value! how am i supposed to deal with this in LINQ? i need the actual value on the server-side in order to be able to run the query as the user expects?

i appreciate any feedback you can offer me.
my application allows users to write dynamic LINQ expressions inside a editor against an in-memory DataSet. here is an example user-defined expression:

from row in pDSet.Tables["XXX"].AsEnumerable()
select new
{
SiteID = row.Field<int>("SiteID"),
Gender = row.Field<string>("Gender")
}

when the user executes their query, i need to copy the results of the query into a new System.Data.DataTable instance.

i know that if i was writing the LINQ query during development, that i could do the following to accomplish this task:

DataTable pDTable = new DataTable("Results");
pDTable.Columns.Add("SiteID", typeof(int));
pDTable.Columns.Add("Gender", typeof(string));

foreach (var projection in query)
{
pDTable.Rows.Add(new object[] { projection.SiteID, projection.Gender });
}

however, since the LINQ query is generated dynamically by the end-user, i dont know the fields of their dynamic projection's anonymous type. yet, i still need to be able to copy their results into a new DataTable instance.

i know i could probably use reflection to discover the anonymous type and its fields, but this seems like an inefficient way to accomplish this task and i dont even know how to find the anonymous type since the runtime will generate a random name for it?!

does anyone know how i can accomplish this? your help is greatly appreciated.

thanks,
ben
Fabrice-

thanks for the reply. i had already indeed downloaded, read, and re-read your chapter 14 looking for this solution.

i do see that you reference this same link (an oversight on my part).

what threw me off from your chapter 14 is that in your 'Listing 14.2,' you are *not* accomplishing the same task as in the mentioned link. rather, you are calling 'table.LoadDataRow()' with a new object[] that gets initialized explicitly with the fields of your anonymous type.

your very next statement (after the note) reads 'Early releases of LINQ provided two query operators to perform the same kind of operation - LoadSequence and ToDataTable - but they've been removed in later releases of LINQ, and aren't included in .NET 3.5. But they have been resurrected by Andrew Conrad from Microsoft.'

the bolded portion of this statement is what threw me off because, in my particular case, for some reason i was thinking that the query operator would perform a similar 'foreach' technique, which in the end it indeed does, but it does so after dynamically discovering (via Reflection) the anonymous type's field and property infos.

this was a mistake on my part. it wasnt clear to me how these query operators worked (reflection). an additional sentence to the affect of 'these query operators will dynamically, using reflection, discover the resulting DataTable's columns and row value's from your projected type's fields and properties...' would have put 2 and 2 together in my mind.

at this point, it is obvious to me that i should inferred this from your text because there isnt really any other way (that i can think of) that the query operators could possibly be implemented. but i just didnt see it and that listing and those few words threw me off.

my apologies and thanks again.
ben
hi Fabrice-

first of all, let me say i loved your book 'LINQ in Action.' i read it cover-to-cover, which is unusual for me. it truely is a great book and thank you for writing it.

unfortunately, your book did not cover the task i needed to accomplish, as described in this thread. rather, it assumes you either have a typed DataSet OR you know the fields of the projected anonymous type - neither of which i have in this situation.

fortunately, i was able to find a solution to this issue today:

http://blogs.msdn.com/aconrad/archive/2007/09/07/science-project.aspx

as a suggestion, it might be useful for you to point your audience to this type of solution in the future. while it is certainly not the 'normal' situation, i think it will come up for lots of developers. it appears from the article that MS discovered this as well. and i definitely believe you will maintain a growing LINQ audience as devs begin to realize the powerful benefits of LINQ and discover your book.

again, thanks for your contribution to the LINQ community.

cheers!
Ben