PubSub Using RabbitMQ with ASP.NET Web API Subscribers

Tags: pubsub, Architecture, MVC, ASP.NET, C#, MVC WebAPI, WebAPI, WCF

A long time ago a colleague of mine, Bhaskar Apparajuvenkata, and I used WCF to implement a publish-subscribe system on our local network. (That was blogged here.) Ok it was less than 5 years ago but given the pace of change since then it feels very antiquated. At the time we were having a hard time convincing our infrastructure team to stand up MSMQ for us. This was before the software-as-a-service movement took hold. Back in those days something like a messaging or service bus framework was seen as an enterprise project. You had to have a project sponsor, huge timelines, and of course a big budget. We were a small development team and our needs were modest so at the time we rolled our own.

Fast forward to today. The system works great in production and we can't imagine life without some sort of message-based architecture. But we're feeling some pain. WCF is overkill. The publisher is a WCF service and subscribers must also be WCF services. We have about a dozen top-heavy subscribers at this point. Some developers just aren't comfortable with WCF configuration or debugging. And testing is tricky. Also our events and subscriber endpoints are stored in a database requiring additional maintenance on our part. So now we're looking at our options for replacing it.

Here are my three main requirements for a replacement solution: (1) simple and lightweight; (2) avoid the need to store and maintain subscriber information; and (3) avoid subscriber long-polling.

Our first obvious option was Azure Service Bus since we already use it with a third-party partner to store transactions for later processing. It's durable, scaleable, simple, and has a rich .NET API. But we have one big problem. Our subscribers are on-prem and because of the firewall we'd have to do long-polling over port 80 to listen for new messages. Azure does provide a broker option called a "service bus relay" that addresses this problem; however it requires WCF bindings to work. That would be déjà vu all over again.

So next up is RabbitMQ. Fast, lightweight, and popular. In just a few hours my colleague stood up a local Ubuntu LTS VM with a RabbitMQ installation. We're doing pubsub so we created a single fanout exchange called "test-exchange." To keep it simple the test-exchange broadcasts its messages to a single queue called "test-queue." For this spike the producer (publisher) is a .NET console app that does nothing but publish the current time. It requires the RabbitMQ.Client nuget package.

Code:
using System;
using System.Text;
using RabbitMQ.Client;

namespace RabbitPublisher
{
    internal class Program
    {
        private static void Main(string[] args)
        {
            var factory = new ConnectionFactory
            {
                HostName = "your.server.domain.com",
                Port = 5672,
                UserName = "square",
                Password = "widget",
                VirtualHost = "/"
            };

            using (var connection = factory.CreateConnection())
            {
                using (var channel = connection.CreateModel())
                {
                    channel.ExchangeDeclare(exchange: "test-exchange", 
                        type: "fanout", 
                        durable: true);

                    var message = "Current time: " + DateTime.Now.ToLongTimeString();
                    var body = Encoding.UTF8.GetBytes(message);

                    channel.BasicPublish(exchange: "test-exchange",
                        routingKey: "",
                        basicProperties: null,
                        body: body);
                }
            }
        }
    }
}

The above publisher code is really a snippet that can be placed in any application that raises events for other applications to handle. This is the essence of message-driven architecture of course. The real problem for us to solve was the subscriber. Most blogs and RabbitMQ tutorials all show console apps. Not very helpful in a production scenario. I thought about it being a Windows Service. But they're a pain to debug and deploy.

How about ASP.NET RESTful Web API? In our shop we're leaning in a microservices direction and for us that means smaller Web API apps. They're easy to write, debug, and deploy. Devs understand HTTP and no complicated WCF configuration required. Here's what I did and it's remarkably simple...

I spun up a new ASP.NET Web API solution in Visual Studio. Then I added the RabbitMQ.Client nuget package. I added a folder named Bus and added this class to it:

Code:
using System;
using System.Diagnostics;
using System.Text;
using RabbitMQ.Client;
using RabbitMQ.Client.Events;

namespace RabbitSubscriber.Bus
{
    public static class MessageListener
    {
        private static IConnection _connection;
        private static IModel _channel;

        public static void Start()
        {
            var factory = new ConnectionFactory
            {
                HostName = "your.server.domain.com",
                Port = 5672,
                UserName = "square",
                Password = "widget",
                VirtualHost = "/",
                AutomaticRecoveryEnabled = true,
                NetworkRecoveryInterval = TimeSpan.FromSeconds(15)
            };

            _connection = factory.CreateConnection();
            _channel = _connection.CreateModel();
            _channel.ExchangeDeclare(exchange: "test-exchange", 
                type: "fanout", 
                durable: true);

            var queueName = _channel.QueueDeclare().QueueName;

            _channel.QueueBind(queue: queueName,
                exchange: "test-exchange",
                routingKey: "");

            var consumer = new EventingBasicConsumer(_channel);
            consumer.Received += ConsumerOnReceived;

            _channel.BasicConsume(queue: queueName, 
                noAck: true, 
                consumer: consumer); 
        }

        public static void Stop()
        {
            _channel.Close(200, "Goodbye");
            _connection.Close();
        }

        private static void ConsumerOnReceived(object sender, BasicDeliverEventArgs ea)
        {
            var body = ea.Body;
            var message = Encoding.UTF8.GetString(body);
            Debug.WriteLine("{0}", message);
        }
    }
}

The plumbing is very similar to the producer code. But notice the ConnectionFactory has two additional properties to recover from a network failure. This is very important to configure because something will go wrong and connections will be interrupted. In my testing I simulated a network outage on the subscriber server itself. We also took down the whole RabbitMQ server and brought it back up. Through both outages the connection was restablished with no problem. The Received event is handled by the delegate function ConsumerOnReceived. It just writes the message body out to the debug output window. In a real production app you'd want to discover what message you received and handle it. This is where a Gang of Four Builder pattern would come in handy. [Update: See Part 2 in this series for a real-world messaging example.]

With the MessageListener class in place we need a way to start it up and keep it running through the application lifecycle of the ASP.NET app. When the app pool gets recycled or if the AppDomain goes down for any reason we need to start it up again. And we want the MessageListener to tear down its connection to the queue gracefully. The bootstrap code to do this gets added to Global.asax.cs:

Code:
using System.Web.Http;
using System.Web.Mvc;
using System.Web.Optimization;
using System.Web.Routing;
using RabbitSubscriber.Bus;

namespace RabbitSubscriber
{
    public class WebApiApplication : System.Web.HttpApplication
    {
        protected void Application_Start()
        {
            AreaRegistration.RegisterAllAreas();
            GlobalConfiguration.Configure(WebApiConfig.Register);
            FilterConfig.RegisterGlobalFilters(GlobalFilters.Filters);
            RouteConfig.RegisterRoutes(RouteTable.Routes);
            BundleConfig.RegisterBundles(BundleTable.Bundles);
            MessageListener.Start();
        }

        protected void Application_End()
        {
            MessageListener.Stop();
        }
    }
}

Obviously you would get those ConnectionFactory values out of the code and into the Web.config file. And you'd want to set noAck to false and make sure to reply back to RabbitMQ with an ACK so that no messages are lost. But otherwise that's all there is to it.

In a future blog post I'd like to go a little deeper with this spike. I'd like to add message payloads that are a little more interesting than the current time. And I'd like to show how to use a Gang of Four Builder pattern in the ConsumerOnReceived delegate function to handle processing of different kinds of messages. But for now this will have to do.

3 Comments

  • Danmar said

    Thank you, your blog post helped me a lot. I have something similar running a connection on .Net API 2 but I didn't know about
    AutomaticRecoveryEnabled, NetworkRecoveryInterval and the Global.asax.cs change.

    Also my implementation was using a lot of CPU because I had a while loop to keep the connection active. It's working much better now. Thanks!

  • Ranjana Mone said

    I have a web api which is hosted on iis7 and rabbitmq 3.5.6. RabbitMQ and IIS are on the same server.
    When I test the web api from the project code which is listening to the rabbitmq queue on the network server, works fine. But web hosted web api is not listening to the queue. I deleted a default user 'guest' and created a new admin user, using the default exchange. I spent the whole day and still couldn't figure out the issue.

    Is there any suggestions on iis or rabbitmq config.

  • james said

    I'd get RabbitMQ off of your WinOS web server and on an Ubuntu LTE server. Too much can go wrong if they share servers and Rabbit was designed for Linux.

Comments have been disabled for this content.