XStab -- The Gory Details

Table of Contents
I. Preface

I am a UNIX Systems Administrator, not a developer. And while any good administrator is a master of hacked scripts, sitting down and writing a complex, complete, and expandable application is not something we usually do. System Administrators (and Network Administrators for that matter, as the two are generally one and the same) have two primary attributes: Laziness and Paranoia. I have tried to banish these demons in XStab, but it would be against my base nature to deny them fully. As such, I'm sure you'll find Laziness in some places where a more elegant solution is desired. Feel free to patch the code. Paranoia can mostly be seen in the regular expressions. I've never trusted user input, and I never will. Despite these warnings, I feel XStab is easy enough to understand for developers to start writing modules for it, and am looking forward to what will be set forth.

Of course, a project of this depth cannot be created in a void, and there are many who contributed directly to its creation. First, let me thank LiQUiD_X for politely listening (or at least politely ignoring) my non-stop talk about XStab, for lending his keen eye to find those errors which I could not, and for being my primary tester and patiently helping to track down logic errors.

I need to thank Reaper, for all of the graphics you see on this webpage. When I asked for graphics help on the oltl.net forums I was expecting pansy-ass crap to be submitted, but Reaper knows his shit. Thanks for all the purty images :)

To Memnoch, 2Drunk, KillMe, Vlkodlak, Frog, Shinobi, and everyone else I have forgotten: I would not do this if it weren't for you. I would not spend every fucking night and weekend coding if I had nobody to enjoy it with. Thanks for your encouragement, support, and patience.

Happy Fragging.

II. Intended Audience

This document is written primarily to those who desire to write new modules for XStab, for those who want to debug XStab, and for those who simply want to understand how XStab works. This is NOT a programming tutorial, nor is it a perl tutorial, and it is assumed readers are familiar with coding in general, and at least know a little perl. To simply run a released package of XStab suited for a particular game, there is no need to know HOW XStab works, and instead should consult the documentation for installing that particular release.

This document is NOT a tutorial, and it is NOT meant as a replacement for reading through pre-existing module code, as this code should serve as your primary example to creating new modules. What this document is are my more subtle notes and concepts about XStab, which can help you understand how everything works together. I make no claim that this documentation is complete. Any help in cleaning it up, adding to it, and making it better would be highly appreciated.

If you are writing a module and would like help, have questions, etc, please email me at dranok@users.sourceforge.net, and I'll see what I can do, or visit http://forums.oltl.net and go to the XStab forums. I'll do everything I can to help you complete your module.

III. Conventions

I have used a few conventions in the creation of XStab. These will be discussed more later but for now you should be aware of the following: Methods starting with an _ (underscore) are intended to be private -- of course, perl doesn't particularily give a swimming hoot about the notions of public or private, but by explicitly specifying which methods are private in this way it is easier to know which ones are public, and therefore likely required as part of the API.

Often its useful to pass a string separated by an identifier. This can be passed by normal methods such as "foo,bar,blah" or "foo:bar:blah", but sometimes this is not sufficient as foo or bar may contain the separator in them: "foo,bar,bl,ah" would be split as 'foo', 'bar', 'bl' and 'ah' instead of 'foo', 'bar' and 'blah'. Also, such a string may be created in one module and passed to a different one, which would then need to know how to split the string. XStab's answer is to use a global separator string, which is currently ':x:'. This is represented by $Global{sp}. The forementioned string would thus be set to "foo$Global{sp}bar$Global{sp}bl,ah". This is messier and less readable than the previous examples, but has many benefits, and its use is required in the API.

In the root XStab directory there is a script called 'build_config.pl'. This will traverse through the modules you include from the 'xstab' file, finding which %Global values are used, and placing them into a config file. If you write a module which requires new config settings, you need to add these to the hash at the top of build_config (view build_config.pl for more info).

%Players is obviously the main storage hash, which is then subdivided into $Players{ClientID}. Since XStab is a real-time bot, it is assumed there is some kind of numerical (usually) id for those players currently connected. This is, for example, the number shown as 'slot' in a status in quake-like games. This is NOT a player's unique identifier, which is also required, and is a set field in the config file. For example, RTCW uses GUIDs, which are stored as $Player{$client}{guid}. Games which do not have a unique ID such as this will have to use something else, like the player's name, which is much less reliable to track players with. The client number is simply a convenient method of identifying currently playing records. It is not 100% necessary to use the clientid in your game module; you could code around the concept of $Players{$guid} -- however, this requires passing a longer string to any subfunctions (and 'client' is almost always passed), and also increases the risk of running out of ram from non-deleted but unused $Players{$guid} records (for example, XCore removes hashes when it receives a ClientDisconnect in the log, but if a player crashes their record won't be deleted). The client id method doesn't suffer from this problem, since, given a 20 player server, you only have 20 slots, or 'ClientIds', thus when a new player connects to slot 19, you know you can safely delete the old player record (XCore does this a different way -- only when the GUID doesn't match the old GUID -- this is because ClientConnect happens every round, whereas the shrub-specific uConnect only happens once when a player connects).

The other important thing is constructor names. I don't like using new() as a constructor, and so have used create(). Since XStab needs to create these objects without knowing anything about them, it would be ill advised to make your own constructor require any explicit arguments. Keeping with the name convention and calling your package constructors create() is strongly advised.

As for variable names, I've never particularly liked any given styles, and since almost all variables are either part of the %Players hash in XData.pm or local to a package, there's no true need to enforce naming rules. That said, there are a few hash prefixes which have special meaning. Hash elements starting with XS_ are XSerialize fields. This is best illustrated with an example. Take RTCW: $Players{client}{mod_kills}{MOD_BLAH} = 5; to store this in a key:value way (and make it readable to webpages/whatever needs to access this), the easiest solution is to convert this into "MOD_BLAH:5,MOD_FOO:3,...". So XS_mod_kills in this case would mean XSerialize mod_kills. It will make more sense as you view the current modules

The following applies only to XDbiMySQL, which is most likely the Storage module you will be using. I place this info here as it should be carried over into any other Storage modules which are written for sanity.

Like XS_, XA_ means append, which will be covered in the Storage module section. NOTE: If you do this, your database rows will become HUGE, which is why I don't use this in XCore or XShrub -- you will need to write some kind of opitimzation script which runs off a cron to convert this giant list into normal serialization fields. But the support is there, should you want it.

There is also XI_, which simply means to ignore this field. Fields starting with XI_ should not be loaded and should not be saved. This is useful when your append field (XA_) is parsed on a cron and placed in better format into a different column. In this case, XStab doesn't need to give a flying hoot about this new column, so by prefixing it with XI_ you can tell XStab this info is irrelevant.

Finally, hash fields prefixed with r_ are intended to be reset often, and used as short-term counters. In fact, XCore and XShrub delete all the r_ fields at the end of each round. As you might have guessed, r_ stands for "round statistic"

IV. Documentation

The real work of XStab is done inside the modules. Wherever possible, modules should be completely agnostic and not care about what other modules are being used, or aren't being used. There are, of course, exceptions to this. XStab uses seven types of modules. They are:

The literal xstab file uses all of these modules with the exception of the command module, which is specific to the game module. In other words, the 'use XSTAB::GAME::Command.pm' or whatever is in the game module, not the main module. Nothing other than the game module has use for the command module, and the only reason to separate it into its own little module is to simplify the coding process. In practice a game module should always be distributed with its command module. For XCore and XShrub there's even a simple API between these modules, so that printing text to a server is simplified with the _do_say method, and adding commands to the voice queue is simplified with the _do_cmd method. More on that later.

As I mentioned, not all modules are purely agnostic. Here is a list of these exceptions:

The command module requires to know about the game module, and the game module needs to know at least what the command module's API is. For example, XCore uses XCoreCommands, while XShrub uses XShrubCommands. Someone could write an XShrubCommands_alt, and XShrub wouldn't complain, as long as the API XShrub uses was followed. These two modules are reliant on each other, as can be imagined.

The sensor modules are completely agnostic, but the game module can require certain modules to be present. For example, using RTCW::XCore, both the XTail and XUdp module need to be present, and any other sensors would be ignored. When you think about it, this makes rational sense -- the sensors don't need to know what they're sending data to, or even what the data is -- but the game module which depends on this data better have a fairly good idea of what it is.

The Voice module is completely agnostic, and while the game module assumes it knows what the Voice module is doing, this isn't always the case. You could swap the XRcon voice module for an XIrc one, for example, although it would be somewhat amusing seeing "chatclient help" statements being sent to an IRC channel *grin*.

The Storage module is also completely agnostic, assuming the game module follows a key:value sequence. While the Debug module has support for nested hashes (up to a point), for simplicity the storage modules and game modules are advised to keep any data which must be stored/loaded to two levels. The first level can be stored as a simple key:value, while the second can be stored as key:XS_value (using the XSerialize method in XData). The game module need know nothing about the storage module. It would be humours to write XRandom, where stored data is sent to /dev/null, and loaded data is randomly generated.

The Debug module is highly agnostic, and only really cares about the Players hash and the Global hash. Debug does need one method implimented in the game module (get_status), but other than that, it needs to know absolutely zilch about the other modules. Likewise, the game module shouldn't even be aware a debug module exists.

The Data module should never, ever, every be changed. It contains global functions (like do_log), and the global hashes shared between all the other modules. It is agnostic in the sense that it's simple, stupid, and only a package because it is easier that way than storing the data in the xstab file itself. You shouldn't think of the Data module as a module which is swappable, as all it contains are data structures and very simple functions.

In order to write your own modules for XStab, you simply need to re-create the public methods for the type of module you wish to write. Click on a module, and view its public methods. Anything else is up to you. As long as your public methods accept the same arguments, and return in the same method, your module should work seamlessly with XStab.

IV.0. XStab

 IV.i. The Game Modules

The Game Module is the most important aspect of XStab from a user's perspective. Note that due to the style I coded XCore and XShrub in, deriving classes requires a bit of copy-and-paste (the data hases aren't inherited). Feel free to design better OOP code when you write your own modules. IV.ii. The Command Modules

The Command module is ONLY used by the Game Module, if you like you don't need to make this its own module, and instead only use the Command module. XStab doesn't care. Like XCore and XShrub, deriving classes requires a bit of copy-and-paste (the data hases aren't inherited). Feel free to design better OOP code when you write your own modules. IV.iii. The Sensor Modules

 IV.iv. The Voice Module

IV.v. The Storage Module

IV.vi. The Debug Module

IV.vii. The Data Module

V. Notes, Tips, and Caevats

The included XStab modules try to be as efficient as possible, and wherever possible are coded to be fully multiplexing. Be careful of writing any module which has a chance to sit there blocking, waiting for input, as this will effectively bring XStab crawling to a stop. You need to remember control gets passed to the game module once every main loop, and the game module should attempt to return control to XStab ASAP. When you're logged into the debug module, you can see how many iterations of the main loop per second XStab is achieving. As long as this is above 30ish, you should be OK. If this drops below 20, the bot will start appearing lagged, and if it drops below 10 something is horribly wrong.

The one module which unfortunately can't be as efficient as the others is the Storage module. The storage module needs to save the current state of the %Players hash, or it needs to load player data (and basically has to wait here). As most database servers will be ran local, this probably isn't that big of a deal, but its certainly something to be aware of.

Should the Storage module become an issue, the following advice could be used to speed things up. Loading players could be interrupted should the process take to long, at which point the %Players{$client} hash should be deleted, thus starting the load process over again (as if a new player joined). This may become inconvenient. Assuming you have CPU/memory to spare, an easy solution to save latency would be to fork() before you transfer control to the Storage module. Be careful with this method, and realize there are reasons I do not do this in the core modules, but instead allow the Storage module to destroy XStab in a worst case scenario. The alternative, which is what I will run on SmokeHerb, is as follows: The Storage module uses blocking IO (unlike the rest of XStab), and so the database queries will block indefinately, effectively halting XStab at that point. After a time, the Storage module will issue the die command, and XStab will terminate. This is its intended behavior. On a server like SmokeHerb, player records are everything, and insuring absolute integrity is more important than the bots other functions. As such, this method is acceptable. On a server where player records are less important, this can be changed.

It is not necessarily to use one of every module, if not doing so makes sense. For example, should you not want the extra overhead of the debug module (not that its very much) You could simply write a Debug module which returns immediately on its public method of do_one_loop. Making dummy classes as this is prefered to directly editing xstab. Same goes for the storage module -- if you never intend to save/load player records, feel free to write a stupid class which just returns immediately on all of its public method calls.

VI. History

I remember the first time I played RTCW. It was the first mptest, and I couldn't figure out who were Axis and who were Allies. I thought the game sucked. I wasn't really into FPS games (not that I am now), but RTCW had a way of growing on me. I bought two copies the day it came out.

But RTCW had some serious problems, problems commonly known as 'wankers'. Yes, these were the boys who couldn't actually *play* the game, so instead took pleasure in killing their teammates. This led more and more servers to become non-TK servers (aka, no friendly fire), which seriously hampered the fun of the game, especially in OLTL, which is my prefered game.

The one day I stumbled upon TQC, and a glimmering and awesome concept was displayed to me. Here I saw one of the earliest versions of WAB, and I knew immediately what it was -- WAB was a simple script reading data from the game's log file and sending data to the RCON system. It was like seeing a windmill for the very first time -- I knew how every component worked, but putting it together in such a way had never crossed my mind.

Of course, my mind thought, I could do it far better. So I tried. And I failed.

Instead of a new bot, a creation known as PotBot briefly entered the world. Storing data in plain-text files, PotBot was merely a few extra methods to Wab.pm and an extra depth array to the main Players hash. I didn't understand WAB too well at that time, and it seemed WAB didn't understand WAB too well, and PotBot lived up to its name and turned out utterly confused, forgetful, and -- well, let's just say it didn't ever actually work right.

We were stupid enough to host the first SmokeHerb.net off of rackspace, which while a fairly decent colocation company, are NOT meant for individuals. I gave up on PotBot, and started mindlessly screaming at any who would utter its name, and indulged myself in EverQuest instead. SmokeHerb.net was shut down, the dream indefinately on haitus.

But EverQuest has a way of eating your brain, and when late summer came I was itching to create something new. I had read several new books on Perl, and had explored the concets of multiplexing and non-blocking IO. Thus, when a prick from a small clan named |COOL| asked me to create a stats system for him, I jumped at the possibility. From this WStab was born.

WSN was the logical extension to WStab, but by December the entire system was simply too much to maintain. As I write this WStab has been out nearly seven months, and no less than two updates are made per week. The code is a mess. Some of the concepts I had toyed with simply didn't work right. And while on the whole everyone likes WStab I knew I could do better. Even as I was exploring avenues to expand WStab I had started to think of its successor in the back of my mind.

That which would succeed WStab would be required to be modular, for it should be able to work with other games besides RTCW, and it should be expandable for different modes of operation. It should have a detailed trace method. It would require an entire new method of thinking about a simple perl bot.

And from this myriad of thoughts, XStab was conceived.