AzurePlot Archives - Freek Paans

Starving outgoing connections on Windows Azure Web Sites

I recently ran into a problem where an application running on Windows Azure Web Apps (formerly Windows Azure Web Sites or WAWS) was unable to create any outgoing connections. The exception thrown was particularly cryptic:

[SocketException (0x271d): An attempt was made to access a socket in a way forbidden by its access permissions x.x.x.x:80]
   System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) +208
   System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket,
     IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception) +464

And no matter how much I googled, I couldn’t find anything related. Since it was definitely related to creating outgoing connection, and not specific to any service (couldn’t connect to HTTP or SQL), I started to consider that WAWS was limiting the amount of outbound connections I could make. More specifically, I hypothesized I was running out of ephemeral ports.

So I did a lot of debugging, looking around for non-disposed connections and such, but couldn’t really find anything wrong with my code (except the usual). However, when running the app locally I did see a lot of open HTTP connections. Now, I’m not gonna go into details but it turns out this had something to do with a (not very well documented) part of .NET: ServicePointManager. This manager is involved in all HTTP connections and keeps connections open so they can be reused later.

When doing this on a secure connection with client authentication, there are some specific rules on when it can reuse the connections, and that’s exactly what bit me: for every outgoing request I did, a new connection was opened, not reusing any already open connection.

The connections stay open for 100 seconds by default, so if I had enough requests coming in (translating to a couple of outgoing requests each), the amount of connections indeed became quite high. On my local machine, this wasn’t a problem, but it seems Web Apps constrains the amount of open connections you can have.

As far as I know, these limits aren’t documented anywhere, so instead I’ll post them here. Note that these limits are per App Service plan, not per App.

App Service Plan	Connection Limit
Free F1	250
Shared D1	250
Basic B1 1 Instance	1920
Basic B2 1 Instance	3968
Basic B3 1 Instance	8064
Standard S1 1 Instance	1920
Standard S1 2 Instances	1920 per instance
Standard S2 1 Instance	3968
Standard S3 1 Instance	8064
Premium P1 1 Instance (Preview)	1920

I think it’s safe to say that the amount of available connections is per instance, so that 3 Instances S3 have 3*8604 connections available. I also didn’t measure P2 and P3 and I assume they are equal to the B2/S2 and B3/S3 level. If someone happens to know an official list, please let me know.

The odd-numbering of the limits might make more sense if you look at it in hex: 0x780 (1920), 0xF80 (3968) and 0x1F80 (8064).

If you run into trouble with ServicePointManager yourself, I have a utility class that might come in handy to debug this problem.

AzurePlot

For the last couple of months I’ve been working on a little side-project: AzurePlot. Yesterday I put out a big new release, with a lot of features that should make it something that’s useful for others as well. From the readme:

AzurePlot plots metrics from various Azure services. It’s designed to be a better alternative to the native charting/monitoring capabilities of the Azure portal, focusing on usability and performance. It works by accessing the APIs provided by the individual services.

The reason I built it is that I’m kind of disappointed with the Azure portal with regard to its charting/metrics capabilities. Whenever there is a production issue with one of our projects on Azure, gaining insight through the Azure portal is just a big PITA because it’s slow, lacks basic features, etc. Simply, it isn’t usable to do the kind of diagnosis I need to do. So that needed to change.

I’ve used Graphite in the past, and was really happy with that, so at first I wanted to get my Azure metrics into Graphite. That became reality with WadGraphEs. However, to access the data you need to have access to the Azure APIs and that requires uploading a management certificate, giving you wide access to the Azure subscription. I didn’t want that kind of responsibility in my product, so set out to build an intermediate application that you would host yourself, and expose the metrics through an additional API of that application. When building that, I figured it would make sense if that project was also able to render the data. That became AzurePlot.

I’m quite happy how it turned out, currently it can:

Read metrics data from Windows Azure Web Sites, Web/worker roles, VMs and SQL Database
Chart those metrics in a dashboard
Export the metrics through an API for external consumption

To give you an idea what it looks like:

websites

There’s still a lot of work to be done, such as consuming data from more data sources and providing more powerful chart manipulations. I will continue working on that the coming months. Even still, if you’re running on Azure you might find it really useful, so check it out!