Bandwidth Arbitrator - Application discovery (How-to create your own classifier)

Mini How-to Create your own classifier (detect application traffic)

Prerequisite: an up and running arbitrator, and basic Unix Command line skills.

Deep within the arbitrator is a section of code, (thanks to Ethan Sommer and layer7 [see CREDITS] ) that looks at Internet packets. It compares the characters inside the packets with well know patterns. These well known patterns are the signatures of specific Internet applications. The programmers who brought you the bandwidth arbitrator and other Linux programmers working on TC (the kernel shaping tool) are working hard to discover these patterns. Problem is, until we know the pattern of an application there is no way to control it. So the goal of the open source community is to find patterns for every application.

If you already have an arbitrator up and running it is really easy to find new patterns.

Put the arbitrator on a link so that only one user is crossing it, I'll explain why below

Set up an application on the user system. The application you want to learn the pattern for, for example Kazaa.

You need to at least have /etc/arbdefault.conf have one line that will be shaping in order for the rest to work. If you don't have any and are using the default settings in arbdefault.conf then you can go down to the SHAPE APP area and stick in this line:

SHAPE APP 3200 DUMMYAPP dummy

Any legal shaping line will do.

You want to remove this line when done....

If you had to add this line then you will need to stop the arbitrator and then start it up again to take.

Now run the command

/bridge/bridge-utils/brctl/brctl debug my 2

Later to turn off debugging you can do:
/bridge/bridge-utils/brctl/brctl debug my 0

This command puts the arbitrator into sniffer mode, a very verbose sniffer mode. While in this mode all the packets the arbitrator sees, including all the funny characters get dumped to the kernel log /var/log/messages. Its not a very pretty sight, so how do you find the pattern of your application? Read on.

Note: before running the debug command it is useful to clear the kernel log, you can do this by typing

Echo > /var/log/messages

After you have the arbitrator in debug mode you should start a download (assuming your application is some kind of p2p), if you started one before don't worry kill it and start another one.

Well the key here is to only have one application going, one download if you will. If you have more than one going at a time it gets confusing when looking for the pattern. When a download is in progress you usually can spot it pretty easily in the brain table.

/bridge/bridge-utils/brctl/brctl getbrain my (will dump the brain table to a terminal screen)

If your application has a download going one of the entries in the brain table will stick out like a sore thumb (it will have big numbers in the fields for AVG and WAVG), the source and destination IP address for this entry are what you need.

Now that you have the source and destination addresses of the download, you need to edit your kernel message log /var/log/messages with your favorite editor.

Search for the first occurrence of the source address from your brain table in the kernel log. Immediately after the Source address you will likely see a big clump of nonsensical characters, now is where the fun begins! Some place in that dump of characters is pattern that identifies the application so the trick is to find it and then you are done.

Well not exactly, depending on what you find. Many times the pattern changes a bit with each download, for example, one time you might see http:xappname, and the next time you might see http:yappname. So how do you capture that pattern? Well the way we do it is to represent that pattern as a regular expression. For the above example the regular expression to cover it would be

http:*appname

the * in the regular expression acts as a wild card.

So now all that is left to do is to tell the arbitrator to limit on our new pattern (limit when the total bandwidth of all users using this application) You do this by adding a line to the config file.

SHAPE APP 3008 30 MYAPP http:*appname

The line above, demonstrates the exact text of what you would enter in /etc/arbdedault.conf to implement shaping on the application we used in the example. Here is the breakdown of all the fields:

SHAPE tells the arbitrator that this is a shaping command
APP tells the arbitrator to use application shaping
3008 is an identifier the arbitrator will put this number in the port fields of the brain table when it discovers this application (pattern). Make sure the number you choose is the next sequential number (from the entry above)
MYAPP is just a name for humans to read the arbitrator does nothing with it.
http:*appname is the pattern we discovered

Okay so restart the arbitrator, run your application and look at the brain table again. Do you see 3008 in the brain table? If so you've done it, you've discovered a new pattern. Make sure you tell us about it so we can distribute it to all the other arbitrator users.
(not telling is 7 years bad luck)

Note: 20