Automated Form Posting with .NET / ASP.NET’s CURL

At times you’ll encounter an issue where a client, customer or merchant does not provide an endpoint to an application you’re required to use. Basically, the application you’re working with does not have a web service, nor do you have access to connect to the database directly but you HAVE TO get data into the system and the only thing available is a web form on a remote server. 

In PHP, you can utilize CURL to communicate with servers and post forms through code.  In .NET, we have the System.Net namespace which contains a ton of little goodies. It is ASP.NET’s implementation that relates to PHP’s CURL library. The MSDN describes this namespace perfectly:

The System.Net namespace provides a simple programming interface for many of the protocols used on networks today.

I’ve used this namespace many times and I have actually used the System.Net.WebClient class in my C# Google Geocode Class. Which brings me to…

Purpose of the blog entry…

In this example I’m going to show you how to use the HttpWebRequest class to post form data to a website to send a text message to your phone (utilizing the website as the service). I’ve already talked about how to send a text message to your phone from .NET, but this time I want to do it using an online service to demonstrate the use of these two classes.

Note: You need to be aware of how HTTP requests work. You need to know that there are headers being sent, Cookies, Content Types, etc. Without a basic grasp on the the request/response model concepts  you’ll be left scratching your head wondering what the hell is going on behind the scenes.

Getting Started

We first need to identify what headers are being sent. I use Fiddler to do this. Fiddler allows you to view the HTTP request/responses down to a very fine grain of detail.

Then, I’ll go to the web page where I’m trying to make the request and then I’ll submit the form I’m working with. Once the form submission is complete, I’ll look at the headers that are being sent (if any at all). In this case we have some because the site we’re working with is PHP based, while we’re working with .NET. INTEGRATION BABY!!

Here is a screen shot of Fiddler and the items I’m look at. (click to get a bigger shot)

I’m looking for the HTTP Headers that are sent across the wire. I captured those in the red-highlighted area on the screen shot above. Now that I know what URL I need to send to (newsend.php – look on the left hand side of the screen shot). I can then set up my HttpWebRequest object and prepare to send it off.

Enough Jibba-Jabba, Lets see some code!

Please note, this is a Console App that was written to do this. You’d normally separate these concerns out into layers. Its a example app, what more did you want? 🙂 (The entire example app can be downloaded at the bottom of this post).

using System;
using System.Collections.Generic;
using System.Text;
using System.Net;
using System.IO;

namespace HttpRequestExample
{
    class Program
    {
        static void Main(string[] args)
        {
            SendTextMessage();
            Console.WriteLine("Press [ENTER] to quit"); 
            Console.ReadLine(); 
        }

        /// <summary>
        /// Sends a Text Message
        /// </summary>
        private static void SendTextMessage()
        {
            string phoneNumber = "YourPhoneNumber"; // Put your phone number here, player.
            string provider = "tmobile"; // Possible Values: ... (see download for full comment) 
            string message = "Test from example application! DateTime:" + DateTime.Now.ToString();
            string headerVars = String.Format(
                "refer=txt2day&to={0}&from=Your+Email+%28optional%29&provider={1}&message={2}&submit.x=90&submit.y=50", 
                phoneNumber, provider, message); 

            
            HttpWebRequest textMessageRequest =  (HttpWebRequest)WebRequest.Create("http://www.txt2day.com/newsend.php");
            textMessageRequest.Method = "POST";
            textMessageRequest.ContentType = "application/x-www-form-urlencoded";
            textMessageRequest.ContentLength = headerVars.Length;
            AddPostVarsToHeader(textMessageRequest, headerVars);
            
            string responseString;
            using (StreamReader streamReader = new StreamReader(textMessageRequest.GetResponse().GetResponseStream()))
            {
                // The response string has the HTML response (if its HTML)
                responseString = streamReader.ReadToEnd();
                
                /* At this point the Txt Message should be sent. 
                 * Scrape the response to check for a thank you message.
                 * On the Txt2Day.com site, the thank you message looks like this:
                 * Thanks! Your Message Has Been Sent
                 */ 
                Console.WriteLine("Message Sent: {0}", responseString.Contains("Thanks"));
                Console.WriteLine("Note: This message can take a few minutes (to a couple hours from my testing) to arrive."); 
            }
        }


        /// <summary>
        /// Adds the <paramref name="values"/> to the the http headers. 
        /// </summary>
        /// <param name="request">The HTTP Request</param>
        /// <param name="values">The values to add to the headers</param>
        private static void AddPostVarsToHeader(WebRequest request, string values)
        {
            using (StreamWriter streamWriter = new StreamWriter(request.GetRequestStream(), ASCIIEncoding.ASCII))
            {
                streamWriter.Write(values);
                streamWriter.Close();
            }
        }
    }
}

Code Explanation

Setting Up The HttpWebRequest Object

            string phoneNumber = "YourPhoneNumber"; // Put your phone number here, player.
            string provider = "tmobile"; // Possible Values: ... (see download for full comment) 
            string message = "Test from example application! DateTime:" + DateTime.Now.ToString();
            string headerVars = String.Format(
                "refer=txt2day&to={0}&from=Your+Email+%28optional%29&provider={1}&message={2}&submit.x=90&submit.y=50", 
                phoneNumber, provider, message); 

            
            HttpWebRequest textMessageRequest =  (HttpWebRequest)WebRequest.Create("http://www.txt2day.com/newsend.php");
            textMessageRequest.Method = "POST";
            textMessageRequest.ContentType = "application/x-www-form-urlencoded";
            textMessageRequest.ContentLength = headerVars.Length;

The HttpWebRequest object uses the static Create method of its Abstract Base Class, WebRequest.

The  “phoneNumber” variable is the phone number you want to text.

The “provider” variable is the name of the provider. The attached solution has the full list of available providers (there are a ton of them in there).

The variable “headerVars” is actually what is posted to the server when the submit button is clicked. If you look at the format of the string you’ll see “to” and “from” and “provider” and “message”. If you look at the source of the website, you’ll notice that the form elements share thsoe names. That’s not a coincidence. These are the actual POST vars that are posted to the server. In this portion of the code we are rebuilding the Headers.

Since we are sending headers we need to specify the length, which is done by setting the ContentLength of the HttpWebRequest.

Next, we put the values into the request object itself by calling:

AddPostVarsToHeader(textMessageRequest, headerVars);

Which is defined as:

/// <summary>
/// Adds the <paramref name="values"/> to the the http headers. 
/// </summary>
/// <param name="request">The HTTP Request</param>
/// <param name="values">The values to add to the headers</param>
private static void AddPostVarsToHeader(WebRequest request, string values)
{
    using (StreamWriter streamWriter = new StreamWriter(request.GetRequestStream(), ASCIIEncoding.ASCII))
    {
       streamWriter.Write(values);
       streamWriter.Close();
    }
}

Here, we write the values into the request stream itself. Now we are ready to send it off and get a request. Here’s how we do that:

             string responseString;
            using (StreamReader streamReader = new StreamReader(textMessageRequest.GetResponse().GetResponseStream()))
            {
                // The response string has the HTML response (if its HTML)
                responseString = streamReader.ReadToEnd();
                
                /* At this point the Txt Message should be sent. 
                 * Scrape the response to check for a thank you message.
                 * On the Txt2Day.com site, the thank you message looks like this:
                 * Thanks! Your Message Has Been Sent
                 */ 
                Console.WriteLine("Message Sent: {0}", responseString.Contains("Thanks"));
                Console.WriteLine("Note: This message can take a few minutes (to a couple hours from my testing) to arrive."); 
            }

There are many ways to obtain the Response from the Request, but at this point I just want to take the Response Stream and write it to the string. What this is doing is sending a web request to the page we defined (newsend.php) with our form variables placed into the headers, and then after the server processes it, it will return a response, in which we capture it from the response stream.

This stream is actually the HTML that is returned from the “thanks.php” page (step through the code in the debugger and look at the HttpWewbRequest.GetResponse() method. It returns a WebResponse which has all kinds of good info about the response.

With the HTML that is returned we can scrape it to see if the word “Thanks!” is included, which signifies that a message was sent. (On the website, when you get to the thanks.php page after manually submitting a message the web site displays: “Thanks! Your message has been sent”. Or something very similar. I know that the message does contain the “Thanks” word though. 🙂

*Note: This service on the website of sending SMS/Text Messages is not SUPER reliable. I was doing some testing and found that sometimes I’d send a message and I’d get it within 5 seconds, other times it was 5-6 hours  before I got my text message on my phone.

At this point you’ve connected to a web site through code, posted a request that contains variables for the server, and had it perform an action. You then received the page and scraped it to find the overall result.

Download

HttpRequestExample.zip (4 KB)