Uncategorized

Identity based throttling in ASP.NET MVC

0 Comment read

For one of our projects, we recently got sort of DoS’d by one of our client’s own (paying) customer. Somehow the rate of requests coming from this particular user increased about 500x, bringing down part of our system. We’re not sure what exactly happened, it might have been a bug at our side or due to the hardware/software configuration at the specific customer. Anyway, we needed to do something to prevent this problem from happening in the future, whatever the cause. So we came up with the idea of introducing throttling, and specifically throttling based on the ASP.NET username (identity).

The basic idea is that if you, as a specific user, hit our service more than a predetermined number of times within a specific period, you’ll be shown a message that you’re being throttled.

In this post I’ll describe how we implemented this. We also open sourced the solution, which you can get on nuget or the code via GitHub.

Detecting the overload

The first thing we need to do is keep track of the number of incoming request per user. As we’re using ASP.NET MVC, we can easily do this by creating an ActionFilter. At this moment we don’t want to throttle on non-controller/actions, such as static resources. If we did, we could’ve used HTTP Modules.

The ActionFilter basically keeps a the count per user in a ConcurrentDictionary, and increments the counter whenever an authenticated user hits it.

public class UserThrottlingActionFilterAttribute : ActionFilterAttribute{
    ConcurrentDictionary<string,ConcurrentLong> _throttlePerUser = new ConcurrentDictionary<string,ConcurrentLong>();

    public int RequestsPerTimeStep{get;set;}

    public override void OnActionExecuting(ActionExecutingContext filterContext) {
        if(!filterContext.HttpContext.Request.IsAuthenticated) {
            return;
        }
                
        var username = filterContext.HttpContext.User.Identity.Name;

        var counter = _throttlePerUser.GetOrAdd(username,ConcurrentLong.Zero);

        var lastValue = counter.Increment();

        if(lastValue <= RequestsPerTimeStep) {
            return;
        }

        StartThrottling();
    }
}

ConcurrentLong is just a wrapper around a long, which we can increment in a threadsafe way and have a reference to:

class ConcurrentLong {
    long _counter;

    public long Increment() {
        return Interlocked.Increment(ref _counter);
    }

    internal static ConcurrentLong Zero {
        get {
            return new ConcurrentLong();
        }
    }
}

To reset the counts, we create and assign a new ConcurrentDictionary every time a request is made and it’s been longer than TimeStep since the last flush. By doing it as part of the request, we don’t need a separate thread for flushing the data. We place a lock around this part of the code, though I’m not sure its absolutely necessary, but I didn’t feel like hurting my brain over it too much:

public TimeSpan TimeStep{get;set;}
readonly object _lockObject = new object();
DateTime _lastFlush = DateTime.Now;

public override void OnActionExecuting(ActionExecutingContext filterContext) {
    if(!filterContext.HttpContext.Request.IsAuthenticated) {
        return;
    }
                
    FlushThrottleCounterIfNecessary();

    ...
}

private void FlushThrottleCounterIfNecessary() {
    lock(_lockObject) {
        if((DateTime.Now - _lastFlush) < TimeStep) {
            return;
        }

        _throttlePerUser = new ConcurrentDictionary<string,ConcurrentLong>();
        _lastFlush = DateTime.Now;
    }
}

Now that we’re able to detect that a user is overloading the system, the next step is to actually throttle the user.

Throttling the user

To throttle the user we basically used a variation on an answer to a question about throttling in ASP.NET on SO. This solution leverages the ASP.NET caching feature to track throttled users for ThrottleBackoff time:

public TimeSpan ThrottleBackoff{get;set;}

void StartThrottling(string username) {
    HttpRuntime.Cache.Add(
        username, 
        true,
        null,
        DateTime.Now.Add(ThrottleBackoff),
        Cache.NoSlidingExpiration,
        CacheItemPriority.Low,
        null
    );
}

To actually let the user know that it’s being throttled and not continue on to the action, we need to set the ActionResult from within OnActionExecuting:

public override void OnActionExecuting(ActionExecutingContext filterContext) {
    if(!filterContext.HttpContext.Request.IsAuthenticated) {
        return;
    }
            
    var username = filterContext.HttpContext.User.Identity.Name;

    if(HttpRuntime.Cache[username]!=null) {
        var result = new ViewResult {
            ViewName = "Throttling",
            ViewData = new ViewDataDictionary(),
        };

        filterContext.Result = result;
        filterContext.HttpContext.Response.StatusCode = 429; // too many requests
        return;
    }

    ...
}

Here we return a special view “Throttling” which contains the error message. Also we reply with HTTP Status Code 429, Too Many Requests.

Conclusion

That’s pretty much it. If you’re looking for a way to throttle your users in an ASP.NET MVC app, don’t look any further. I put the source on GitHub, it comes with a sample MVC app and a jmeter test script for experimentation.  You can also install it via nuget: Install-Package asp.net-user-throttling.

Some final notes:

  • It should be clear that this is not a complete (D)DoS protection strategy, it’s actually much more a way to protect yourself from someone accidentally overloading your system.
  • Since we’re using action filters, this will only work for URLs that actually map to a controller/action pair.
  • In a multi-node situation, there will be a cache and counter per node, so a user might easily do more requests than the amounts you specify. For us, this wasn’t a particular problem because it’s about orders of magnitude.  More serious though, is that you might be throttled on 1 node and not on the others. This would give an unstable and annoying experience to the users. We solved this by actually setting a cookie that throttling is going on and when that’s present we also reply with the you’re being throttled page.
  • In our solution, we allow the user to override its throttling, also via cookie. This is again because we’re protecting against accidental errors. So if a user is being throttled while doing legitimate work, he can actually ignore the throttling.