TUTORIAL: Writing a Protocol

For this walkthrough, we’re going to create a “time server” (similar to NTP) called PlayTime. So we could create a time server repository and an install script that would put it under the joemamma directory.

<homedir>/Playground/src/joemamma/playtime

To be clear, the repository is just for the PlayTime code, but you could have it install in this sibling fashion format.

Design the Protocol

The next step is to figure out how the protocol is going to work. Ideally, you would generate a PRFC (a Playground RFC; To get a feel for RFC’s, take a look at the original TCP RFC). But for our little demo here, let’s just do a protocol design light.

Our PlayTime server is going to provide a “get time” operation.  For this operation, the client sends a message asking for the time. The only parameter is the time zone. Let’s assume that time zone is in a format of +/- hours from UTC. For now, we’ll represent this network message as GetTime(tz). The actual network message is a string of raw bytes; this is just a way of representing those bytes in design.

The response is the current time along with an md5 checksum to prevent tampering. Mostly to show you how to do it, we’ll take the md5 over the entire packet (instead of just over the time value itself). The time will be recorded as a FLOAT4 (4-byte, floating point) timestamp (as returned by time.time()). We’ll represent this network message asCurTime(timestamp, md5sum).

Accordingly, the protocol looks something like this

[ Client ] ------ GetTime(tz) ------> [ Server ]
[ Client ] <--- CurTime(ts, md5) ---- [ Server ]

Upon further reflection, note that there is nothing tying these two messages together. Let’s modify GetTime to take a nonce, and require CurTime to include that nonce in the response:

[ Client ] ------ GetTime(nonce, tz) ------> [ Server ]
[ Client ] <--- CurTime(nonce, ts, md5) ---- [ Server ]

It’s a very simplistic protocol. It doesn’t deal with errors (e.g., if the timezone delta is too big or otherwise incorrect) for example. But it’s a reasonable protocol to implement as a test.

Design the Network Messages

In our design, we specify the sending and receiving of network data. In real network operations, you have to serialize (i.e., copy data out of program structures into a raw bytes of text) before transmission and deserialize (i.e., copy data out of raw bytes of text into program structures) after reception. Playground, however, makes this easy.

To design a message, you simply declare a class that has a PLAYGROUND_IDENTIFIER, a VERSION, and a BODY.

 

  • PLAYGROUND_IDENTIFIER – A string that uniquely identifies the message. It can be any string, but I recommend using a hierachical naming system similar to a java or python package. For example: “playground.<company name>.<product>.<message name>”
  • MESSAGE_VERSION – A version on the message to recognize incompatibilities. For example, if we released a later PlayTime protocol with updated messages, we would update the version. This would prevent inadvertently attempting to communicate with out-of-date messages.
  • BODY – a list of tuples of data fields. These are the actual storage fields in the message

This is best explained by example. So, we have two message types, “GetTime” and “CurTime”. Let’s create the first one


from playground.network.message.ProtoBuilder import MessageDefinition
from playground.network.message.StandardMessageSpecifiers import INT1, UINT8

class GetTime(MessageDefinition):
  PLAYGROUND_IDENTIFIER = "playground.joemamma.playtime.GetTime"
  VERSION = "1.0"
  BODY = [
    ("timezone", INT1),
    ("nonce", UINT8)
  ]

 

This class defines a message that has two fields: a timezone of type INT1 (signed integer, 1 byte) and a nonce field of type UINT8 (unsigned integer, 8 bytes).

The definition of the message must exist on both the sending and receiving playground nodes, otherwise it won’t deserialize correctly. When we write PRFC standards documents, they EXPLICITLY identify the identifier of the message, its version, and its contents.

The other message is similarly defined:


from playground.network.message.ProtoBuilder import MessageDefinition
from playground.network.message.StandardMessageSpecifiers import FLOAT4, STRING

class CurTime(MessageDefinition):
  PLAYGROUND_IDENTIFIER = "playground.joemamma.playtime.CurTime"
  VERSION = "1.0"
  BODY = [
    ("timestamp", FLOAT4),
    ("nonce", UINT8),
    ("md5", STRING)
  ]

 

Implement Client and Server Prototypes

Let’s implement the functionality a little at a time. We will leave some fields unimplemented and un-used as we iteratively create our code.

Let’s start out by not using the nonce, timezone, or md5 fields. We’ll simply send a message requesting time and get a response back in localtime

Our Server code will be very simple. The Playground framework provides the SimpleMessageHandlingProtocol as an easy way to write a networking code. This class, a subclass of Playground’s Protocol class, can register message types with handlers. When a message is received over the network, it routes it to the correct handler. Our server is designed to process only one type of protocol: GetTime.


from joemamma.playtime import GetTime

class PlayTimeServer(SimpleMessageHandlingProtocol):
  def __init__(self, factory, addr):
    # call base class constructor
    SimpleMessageHandlingProtocol.__init__(self, factory,addr)
    # register a handler for the GetTime network message
    self.registerMessageHandler(GetTime, self.__handleGetTimeMessage)

 

I’m assuming that the GetTime message we defined above is accessed from the “joemamma.playtime” module. It doesn’t matter where it is, but I like to have it in a location that mirrors the hierarchical naming in the PLAYGROUND_IDENTIFIER.

The PlayTimeServer class inherits from SimpleMessageHandlingProtocol and, like many object oriented languages, you have to explicitly call the base class’s constructor. The base class requires the factory that created the protocol and the playground address as arguments. We’ll talk more about the factory in a bit. For now, it isn’t really used.

The method “registerMessageHandler” is pretty obvious. It takes a message definition (GetTime in this case) and registers a handler. When a GetTime message is received, the handler will be called. Let’s write that handler now.


from joemamma.playtime import CurTime, GetTime
from playground.network.common.MessageHandler import SimpleMessageHandlingProtocol
import time

class PlayTimeServer(SimpleMessageHandlingProtocol):
  def __init__(self, factory, addr):
    # call base class constructor
    SimpleMessageHandlingProtocol.__init__(self, factory,addr)
    # register a handler for the GetTime network message
    self.registerMessageHandler(GetTime, self.__handleGetTimeMessage)

  def __handleGetTimeMessage(self, protocol, msg):
    curTimeMessageBuilder = MessageData.GetMessageBuilder(CurTime)
    curTimeMessageBuilder["timestamp"].setData(time.time()+time.timezone) # +timezone for utc
    self.transport.writeMessage(curTimeMessageBuilder)
    self.transport.loseConnection()

Our handler, __handleGetTimeMessage takes two argument: protocol and msg. The protocol is just the instance of the calling protocol and, in this case, is identical to self (it is included because the handler need not be a method; if it’s an external function, it will need the reference to protocol for access). The variable msg is a MessageBuilder, a class for creating instances of message types.

Notice that we do not even use msg in this first version. Our handler is only for messages of type GetTime. We already know that’s being requested. So we simply create a new response message and send it.

To create a response message of type CurTime, we use MessageData.GetMessagebuilder(CurTime). The MessageBuilder class makes it easy to set the values of the data before serialization. The transport.writeMessage() method turns the MessageBuilder into raw bytes and sends it over the network.

The call to loseConnection() closes down the protocol, which is a “one-shot” kind of thing. It doesn’t leave the connection over after responding with the time.

Our client protocol is only slightly more complicated because we have to trigger the actual request for getting the time.


from joemamma.playtime import CurTime, GetTime
from playground.network.common.MessageHandler import SimpleMessageHandlingProtocol

class PlayTimeClient(SimpleMessageHandlingProtocol):
  def __init__(self, factory, addr):
      SimpleMessageHandlingProtocol.__init__(self, factory, addr)
      self.__curTime = None
      self.registerMessageHandler(CurTime, self.__curTimeHandler)

  def connectionMade(self):
      self.__sendGetTimeRequest()

  def __sendGetTimeRequest(self):
      getTimeMessageBuilder = MessageData.GetMessageBuilder(GetTime)
      self.transport.writeMessage(getTimeMessageBuilder)

  def __curTimeHandler(self, protocol, msg):
      msgData = msg.data()
      self.__curTime = msgData.timestamp

The method connectionMade is a callback that is triggered after the network connection has been established. Once that happens, we can request the time and we immediately create a message of type GetTime and send it over the network. Note that we didn’t set any fields, because this early prototype doesn’t use them.

We also take the CurTime message, when received, and get the timestamp. Note that the received msg is a MessageBuilder, but calling the data() method converts it into a simple structure holding the data of the appropriate field.

You still would need to modify the PlayTimeClient to provide the time data to other code. This can be done with callbacks and a special type of Twisted class called Deferred. We won’t do that in this lesson. If you want to see the received timestamp, just print it to the screen when received.

Test The Prototypes

The code above is fully operational. Let’s set it up for use with our PlaygroundNode infrastructure.

First, we need to create Factories. A Factory is used to spawn protocol instances. Take the PlayTime server. It could receive requests from many clients at the same time, each one would need its own instance of the PlayTimeServer protocol object. The Factory creates each instance every time a new connection comes in. We also use a Factory for the client, even though there is only one.

Fortunately, creating a simple factory is very easy.


class PlayTimeServerFactory(ClientApplicationServer):
    Protocol=PlayTimeServer
    
class PlayTimeClientFactory(ClientApplicationClient):
    Protocol=PlayTimeClient

To repeat, the factory will create new instances of your protocols every time one is needed.

Now, to make this work with PlaygroundNode, we need to define a start() and stop() method, along with a Name variable for our module. I prefer to put these in a class as shown below. Note that I call my class PlaygroundNodeControl, but there’s nothing special about this name. It can be called anything.


class PlaygroundNodeControl(object):
    Name = "PlayTime"
    def __init__(self):
        # setup here if necessary
        pass

    def start(self, clientBase, args):
        self.clientBase = clientBase
        result, msg = True, ""
        mode = args[0]
        if "server" == mode:
            playTimeServer = PlayTimeServerFactory()
            result = clientBase.listen(playTimeServer, 222)
            if result == False:
                msg = "Could not start server"
        elif "client" == mode:
            playTimeServerAddr = playground.network.common.PlaygroundAddress.FromString(args[1])
            playTimeClient = PlayTimeClientFatory()
            Timer.callLater(0, lambda: clientBase.connect(playTimeClient, playTimeServerAddr, 222)
        else:
            result, msg = False, "requires either 'server', 'client'"
        return result, msg
    
    def stop(self):
        self.clientBase.disconnectFromPlaygroundServer()
        return True, ""
control = PlaygroundNodeControl()
start = control.start
stop = control.stop
Name = control.Name

In start up, it takes either a server or client mode as an argument. In the server case, it starts the server listening on port 222.  Otherwise, it sets up a callback to start the client connecting to the server.

Again, the module needs the start, stop, and Name to be defined globally with respect to the module. So, we instantiate the PlaygroundNodeControl and copy its attributes. You should now be able to launch the playground server and client and test the basic functionality.

Expand the Client and Server Functionality

Now that we have the basic functionality working, we can go back and expand our client and server to their full specifications.

Let’s start by updating the server to handle the timezone and nonce.


from playground.network.common.MessageHandler import SimpleMessageHandlingProtocol
import time

class PlayTimeServer(SimpleMessageHandlingProtocol):
  def __init__(self, factory, addr):
    # call base class constructor
    SimpleMessageHandlingProtocol.__init__(self, factory,addr)
    # register a handler for the GetTime network message
    self.registerMessageHandler(GetTime, self.__handleGetTimeMessage)

  def __handleGetTimeMessage(self, protocol, msg):
    msgData = msg.data()
    hoursDelta = msgData.timezone # for example, +1 or -5 hours
    secondsDelta = hoursDelta * 3600
    curTimeMessageBuilder = MessageData.GetMessageBuilder(CurTime)
    curTimeMessageBuilder["nonce"].setData(msgData.nonce) ## Copy the nonce
    curTimeMessageBuilder["timestamp"].setData(time.time() + time.timezone + secondsDelta)
    self.transport.writeMessage(curTimeMessageBuilder)
    self.transport.loseConnection()

Similarly, we need to modify the client to send the timezone data. We need to make a few changes and also modify the PlaygroundNodeControl to accept a timezone argument.


class PlayTimeClient(SimpleMessageHandlingProtocol):
  def __init__(self, factory, addr):
      SimpleMessageHandlingProtocol.__init__(self, factory, addr)
      self.__curTime = None
      self.__nonce = random.randint(0,2**64)
      self.__factory = factory
      self.registerMessageHandler(CurTime, self.__curTimeHandler)

  def connectionMade(self):
      self.__sendGetTimeRequest()

  def __sendGetTimeRequest(self):
      getTimeMessageBuilder = MessageData.GetMessageBuilder(GetTime)
      getTimeMessageBuilder["timezone"].setData(self.__factory.TimeZone)
      getTimeMessageBuilder["nonce"].setData(self.__nonce)
      self.transport.writeMessage(getTimeMessageBuilder)

  def __curTimeHandler(self, protocol, msg):
      msgData = msg.data()
      if self.__nonce != msgData.nonce:
          # Handle error?
          pass
      self.__curTime = msgData.timestamp

class PlayTimeClientFactory(ClientApplicationClient):
    Protocol=PlayTimeClient
    TimeZone=0

class PlaygroundNodeControl(object):
    Name = "PlayTime"
    def __init__(self):
        # setup here if necessary
        pass

    def start(self, clientBase, args):
        self.clientBase = clientBase
        result, msg = True, ""
        mode = args[0]
        if "server" == mode:
            playTimeServer = PlayTimeServerFactory()
            result = clientBase.listen(playTimeServer, 222)
            if result == False:
                msg = "Could not start server"
        elif "client" == mode:
            playTimeServerAddr = playground.network.common.PlaygroundAddress.FromString(args[1])
            timezone = int(args[2])
            playTimeClient = PlayTimeClientFatory()
            playTimeClient.TimeZone = timezone
            Timer.callLater(0, lambda: clientBase.connect(playTimeClient, playTimeServerAddr, 222)
        else:
            result, msg = False, "requires either 'server', 'client'"
        return result, msg

Note that we accept another parameter from the PlaygroundNode that is a timezone, and we use it to set a class variable in the PlayTimeClientFactory. That variable, TimeZone, is used to set the value of the packet transmitted.

We also generate a random number as a nonce and transmit it as well. If the returned packet doesn’t have our nonce, we should have some kind of error.

Handling MACs on a Message

The final bit of code is the most complicated. We want to generate a MAC (Message Authentication Code) on our packet. But we haven’t ever actually seen our actual raw bytes of packet. We’ve only worked with an interface to it.

Recall that we want to create an MD5 sum on the packet. The challenge, of course, is that the md5 sum field is part of the packet. How can we generate a MAC over a field that isn’t set yet?

In typical network protocols, the common solution is to set the field to 0, create the MAC, and then write over the 0 bytes. Here is some sample code:

class PlayTimeServer(SimpleMessageHandlingProtocol):
  def __init__(self, factory, addr):
    # call base class constructor
    SimpleMessageHandlingProtocol.__init__(self, factory,addr)
    # register a handler for the GetTime network message
    self.registerMessageHandler(GetTime, self.__handleGetTimeMessage)

  def __handleGetTimeMessage(self, protocol, msg):
    msgData = msg.data()
    hoursDelta = msgData.timezone # for example, +1 or -5 hours
    secondsDelta = hoursDelta * 3600
    curTimeMessageBuilder = MessageData.GetMessageBuilder(CurTime)
    curTimeMessageBuilder["nonce"].setData(msgData.nonce) ## Copy the nonce
    curTimeMessageBuilder["timestamp"].setData(time.time() + time.timezone + secondsDelta)
    curTimeMessageBuilder["md5sum"].setData("\x00"*16) # this creates 16 bytes of 0

    rawPacketBytes = Packet.SerializeMessage(curTimeMessageBuilder)

    md5sumBytes = md5.md5(rawPacketBytes).digest()
    curTimeMessageBuilder["md5sum"].setData(md5sumBytes)
    
    self.transport.writeMessage(curTimeMessageBuilder)
    self.transport.loseConnection()

Verifying the mac is left as an exercise for the reader.

Summary

In this example, we’ve walked through every stage of creating a Playground program. From this, you should begin to get comfortable designing and implementing your own protocols.

Leave a Reply

Your email address will not be published. Required fields are marked *