Friday, March 6, 2009

Data storage

Contemporary Database access.

Current database technology is mostly based on the relation database, a huge peice of expensive black box software.

However most people design their data from object oriented classes. On all web sites I've worked on, we always first designed the .net data classes, then created tables in the database, and then a mapping between the. It don't really matter if this mapping is hibernate, linq2sql or some manual mapping. And the performance bottlenecks always seems to be in the databases too, even if there is only a few hundred mb of data.

What i would like to see is two things.
An ORM or object database that i don't have to design the database tables to use. Maybe it generates the tables from attributes i put on my classes.

Secondly i want the data layer to help me cache data, writing caching policies is tiresome, and some messaging system between different servers to invalidate the cache should be in here as well.

Some things are happening that may finally put the relation database to rest. SimpleDB and all the other database from the cloud "operating systems" are moving away from relations and complex concurrency models to scale better. I have no clue how to do complex queries and aggregation on these platforms. Maybe we got too spoiled on relation databases and their unlimited query complexities.

Thursday, July 10, 2008

Radchart in ASP.NET MVC

Telerik has a lot of really nice controlls, howevery i douby very many of them will work with the mvc framework. One of the nicest controlls is RadChart, that makes charts, and accordint to easy to get to work.

Inspired by i wanted to make a more elegant approach, with an action rendering a chart without saving it to disk.

To use an action to render a image, we need to make an url to it. We create a helper extension to help us with that.

   public static class ChartResultHelper
       public static string Chart<T>(this HtmlHelper helper, System.Linq.Expressions.Expression<Action<T>> action, int width, int height, string alt)
           where T : Controller
           string url = helper.BuildUrlFromExpression<T>(action);
           return string.Format("<img src=\"{0}\" width=\"{1}\" height=\"{2}\" alt=\"{3}\" />", url, width, height, alt);

and use it in a regular viewpage

       <%= Html.Chart<ReportController>(c => c.Chart(1, 200, 483), 200,483, "Report")%>

Ofcourse some of this could be a parameterized better but shows the point. As you may see, i have mapped this image to the action chart in report controller, lets look at that.

       public ActionResult Chart(long id, int h, int w)
           Telerik.WebControls.RadChart objChart = new Telerik.WebControls.RadChart();
           // the data we want to chart
           Dictionary<string, double> objValues = new Dictionary<string, double>();
           objValues.Add("Svar på foruminnlegg", 4);
           objValues.Add("Nye tråder", 100);
           objValues.Add("Bilder i foruminnlegg", 20);
           // setting some chart parameters
           objChart.Height = h;
           objChart.Width = w;
           objChart.Margins.Right = new Unit(30, UnitType.Percentage);
           objChart.Margins.Bottom = new Unit(1, UnitType.Percentage);
           objChart.Margins.Left = new Unit(1, UnitType.Percentage);
           objChart.Margins.Top = new Unit(1, UnitType.Percentage);
           objChart.Background.MainColor = Color.White;
           objChart.Background.BorderColor = Color.White;
           // creating a series and adding it to the chart
           ChartSeries cSeries = new ChartSeries();
           cSeries.Name = "Poeng";
           cSeries.Type = ChartSeriesType.Pie;
           cSeries.LegendDisplayMode = ChartSeriesLegendDisplayMode.ItemLabels;
           String[] colorArray = new String[7] { "#00839C", "#E15D1F", "#2AB1C4", "#FFAB13", "#D0FAFF", "#CF8300", "#397BB2" };
           int k = 0;
           foreach (var entry in objValues)
               cSeries.Items.Add(new ChartSeriesItem() {Name = entry.Key, YValue = entry.Value, MainColor = ColorTranslator.FromHtml(colorArray[k]) });
           // send the chart to the view engine
           return new ChartResult() { Chart = objChart };

Looks easy enough, just need to create the view engine part of the soulution.

   public class ChartResult : ActionResult
       public Telerik.WebControls.RadChart Chart { get; set; }
       public override void ExecuteResult(ControllerContext context)
           if (Chart == null)
               throw new ArgumentNullException("Chart");
           Image image = Chart.GetBitmap();
           ImageCodecInfo encoder;
           EncoderParameters encoderParams = null;
           context.HttpContext.Response.ExpiresAbsolute = DateTime.Now.AddMinutes(5);
           encoder = ImageCodecInfo.GetImageEncoders().Single(c => c.FormatID == ImageFormat.Jpeg.Guid);
           encoderParams = new EncoderParameters(1);
           encoderParams.Param[0] = new EncoderParameter(Encoder.Quality, 100L);
           context.HttpContext.Response.ContentType = encoder.MimeType;
           image.Save(context.HttpContext.Response.OutputStream, encoder, encoderParams);

Saturday, June 14, 2008

Google app-engine opens for all

Google app-engine opened a while back for a limited number of developers, now however they have silently opened for everyone

So now all i have to do is find a good idea to test it out, and learn Phyton ofcourse.

Automatic single sign in for web applications

While OpenID and probably other emerging technologies are good, they don't really solve the problem i have when developing applications for my employers customers.

Firstly the sign in isn't really automatic, and its not transparent. Having multiple identity providers makes no since in this scenario. This makes the sign in process easier as we know what identity server we will use.

What the new OpenID specs seems to focus a lot on, is delivering profile data from the identity provider to the website. There is a lot of research and work on standards in this field. The reason behind this is the trust issue, identity providers can't trust the website.
However when we want a single sign in on several websites, we do trust all the sites, and the user trusts the sites more than a identity provider.

I guess our problem is a lot more simple than the problems OpenId is trying to solve. All we want is that users signing in on one site, is automatically signed in when they visit one of our other sites.
Since we control all the databases we can easily share profile data with propitiatory web-services.

So the problem really comes down to setting cookies on several domains. This can be done with redirects and images.

Tuesday, June 3, 2008

MVC and OpenId

I've read a lot about mvc and openid the last months, but i haven't had the time to use it for anything yet. One of the problems with mvc as i saw it, was the dependence on the url routing module. IIS 5 and 6 don't like modules in .net, i've tried som ways to get around it, but couldn't make it good enough.

Since the routing module workes without filename extensions i assumed that it wouldn't work. I was wrong however. Just by adding a ".mvc" to the controller names, everything workes fine.

The next step was adding openid. I've been looking for better ways to log in and authenicate users on the web a long time. And openid seemes like nice system. Anyone can use it, there are many id providers, the authenication method is not on your site and even microsoft and google seems to like it.

First part getting and installing mvc preview 3 was easy. Getting it to work with iis5 wasn't as hard as i tougth.

Gave the aspnet and iusr users access to my project folder

chaged the routing to "{controller}.mvc".

           routes.MapRoute("Default", "{controller}.mvc/{action}/{id}",
               new { controller = "Home", action = "Index", id = "" }

And all references to "~/*" to "~/*.mvc".

The Dotnetopenid library has an mvc sample, so maybe the task at hand wasn't that hard afterall.

added this to my config.web

<authentication mode="Forms"><forms defaultUrl="~/Home.mvc" loginUrl="~/User.mvc/Login"/></authentication>

Added the userconrtoller and views/user from the sample

To make the usercontroller work with preview 3 i had to change the "void" to "ActionResult", and change the "RenderView" to "return View("Login");" instead of the old syntax.

       public ActionResult Index()
           if (!User.Identity.IsAuthenticated) Response.Redirect("~/User.mvc/Login?ReturnUrl=Index");
           return View("Index");

This is everything i did to make it work, now i can log in with any of my openid accounts.

In my opinion openid by itsself isn't that usefull, the openid provider needs to give us some more information, like email and nickname the very least. The openid has two ways to do this (or dotnetopenid), we got simple registration extension and attribute exchange.

I couldn't get attribute exchange to work, maybe my providers didn't support it, or i used the api wrong.

I did manage to get som info from the provider with simple registration, but the blogger openid didn't want to share anything.

I'm not sure if this is the current weakness in openid, that these protocols aren't that strandarized and supported yet. I'll do some more research on this soon.

to send request for simple registration fields and recive them i changed the usercontroller.authenitcate() method a bit.

               var openid = new OpenIdRelyingParty();
               IAuthenticationRequest authReq = openid.CreateRequest(Request.Form["openid_identifier"]);
               ClaimsRequest fetch = new ClaimsRequest();
               fetch.Email = DemandLevel.Request;
               fetch.Nickname = DemandLevel.Request;
               fetch.TimeZone = DemandLevel.Request;

requests the email,nickname and timezone data

                       ClaimsResponse fetch = openid.Response.GetExtension(typeof(ClaimsResponse)) as ClaimsResponse;
                       if (fetch != null)
                           string email = fetch.Email;
                           string nickname = fetch.Nickname;
                           string time = fetch.TimeZone;

extracts the data from the response

So what now? we have a mvc site that we can login with openID. We need a authenication and autorsation framework for our pages, so we don't have to write the same code and checks on every controller action. has an excelent atricle on using .net attributes in the code to regulate access to controller actions. It was really easy to get working too, all i did was copy the code for RequiresAuthenticationAttribute and decorate the actions that required an authenicatied user.

       public ActionResult ShowSecretData()
           ViewData["Title"] = "Personal information";
           return View();


dotnetopenid (you'll find links to all the specs here too)

Friday, May 23, 2008

Caching in

I've been using a simple caching system on the project i'm currently working on for some time. But i realized some limitation of this system.

public object GetItemFromCache(int id)
    object o = Cache["item(" + id.ToString() + ")"];
    if (o == null)
        o = GetItemFromDatabase(id);
        Cache.Insert("item(" + id.ToString() + ")",
    return o;

I wanted a more standardized way to define the data being cached. The data I cache are of 2 main types, data about certain users that is accessed many times for a limited time span, and data accessed a lot by all users. This data is either dependent on changes in the database or its not that important that the data is up to date.

The second type of data can be expensive to create, it comes for web services and heavy database queries. A problem with the caching method above is we can have situations where several threads request the data at the same time, doesn't find it and queries the database at the same time. I don't want this to happen, it can be resolved in two ways, either wait on the first thread to get the data, or show the old data until the new is available. And if we store the old data, we might as well automatically fetch updateed data when the cache expires.

Here is the system i came up with. I was a bit inspired by "Implementing Generic Caching", but i found this approach a bit limiting to what objects you could cache. I instead used a different system where we define catchable objects.

public class CachedItemMetadata
    public string BaseKey { get; set; }
    public ObjectLoader Loader { get; set; }
    public bool AutoReload { get; set; }
    public int AbsoluteExpiryMinutes { get; set; }
    public int SlidingExpiryMinutes { get; set; }
    public string[] TableDependency { get; set; }
public delegate object ObjectLoader(object[] param);
public class CachingService
    private class CachedItem
        public Dictionary<string, object> Items { get; set; }
        public Dictionary<string, object> OldItems { get; set; }
        public Dictionary<string, Mutex> Locks { get; set; }
        public Dictionary<string, object[]> Params { get; set; }
        public CachedItemMetadata Metadata { get; set; }
        public CachedItem(CachedItemMetadata i)
            Items = new Dictionary<string, object>();
            OldItems = new Dictionary<string, object>();
            Locks = new Dictionary<string, Mutex>();
            Params = new Dictionary<string, object[]>();
            Metadata = i;
    private static Dictionary<string, CachedItem> items;
    static CachingService()
        items = new Dictionary<string, CachedItem>();
    protected static void Init()
        Add(new CachedItemMetadata()
            AbsoluteExpiryMinutes = 10,
            AutoReload = true,
            BaseKey = "CalculatedStats",
            TableDependency = null,
            SlidingExpiryMinutes = 0,
            Loader = delegate(object[] param)
                return Databroker.CalculateStatistics(
    public static void Add(CachedItemMetadata item)
        items.Add(item.BaseKey, new CachedItem(item));
    public static string CreateFullKey(string key, object[] param)
        System.Text.StringBuilder sb = new System.Text.StringBuilder();
        bool isFirst = true;
        if (param != null)
            foreach (object o in param)
                if (!isFirst)
                    isFirst = false;
        return sb.ToString();
    public static T Get<T>(string key, object[] param) where T : class
        object obj = Get(key, param);
        return obj as T;
    public static object Get(string key, object[] param)
        Cache objCache = HttpContext.Current.Cache;
        string fullKey = CreateFullKey(key, param);
        object obj = objCache[fullKey];
        if (obj == null && items.ContainsKey(key))
            CachedItem objItemInfo = items[key];
            // check that the lock exists
            if (!objItemInfo.Locks.ContainsKey(fullKey))
                lock (objItemInfo.Locks)
                    if (!objItemInfo.Locks.ContainsKey(fullKey))
                        objItemInfo.Locks.Add(fullKey, new Mutex());
            Mutex objLock = objItemInfo.Locks[fullKey];
            bool didLock = objLock.WaitOne(0, false);
            if (!didLock)
                // if autoreloaded item try and see if we can return the old item
                if (objItemInfo.Metadata.AutoReload 
                    && objItemInfo.OldItems.ContainsKey(fullKey))
                    obj = objItemInfo.OldItems[fullKey];
                    if (obj != null)
                        return obj;
                // wait for load to complete
                if (objLock.WaitOne())
                    return objItemInfo.Items[fullKey];
                    return null;
                obj = objItemInfo.Metadata.Loader(param);
                CacheDependency dep = null;
                TimeSpan sliding = Cache.NoSlidingExpiration;
                DateTime abs = Cache.NoAbsoluteExpiration;
                if (objItemInfo.Metadata.TableDependency != null)
                    dep = CreateSqlDependency(objItemInfo.Metadata.TableDependency);
                if (objItemInfo.Metadata.SlidingExpiryMinutes > 0)
                    sliding = new TimeSpan(0, objItemInfo.Metadata.SlidingExpiryMinutes, 0);
                if (objItemInfo.Metadata.AbsoluteExpiryMinutes > 0)
                    abs = DateTime.Now.AddMinutes(objItemInfo.Metadata.AbsoluteExpiryMinutes);
                    new CacheItemRemovedCallback(delegate(string key2, object o, 
                        CacheItemRemovedReason reason)
                        ItemRemoved(objItemInfo, key2, o, reason);
                objItemInfo.Items[fullKey] = obj;
                objItemInfo.Params[fullKey] = param;
        return obj;
    private static void ItemRemoved(CachedItem item, string key, 
        object o, CacheItemRemovedReason reason)
        if (item.Metadata.AutoReload)
            item.OldItems[key] = o;
        if (item.Metadata.AutoReload && item.Params.ContainsKey(key))
            Get(item.Metadata.BaseKey, item.Params[key]);