How I Built a Working Online Poker Bot, Part 3

Introduction

I'm a big fan of pet projects. You know the ones I mean: the projects we love to start and hate to finish. The two-week remodeling gig that takes two years. The '69 Mustang sitting on cinderblocks in the back yard while seasons rotate. The unfinished novel lurking on the nether regions of your hard drive. And for programmers and poker players around the world, a million unfinished tools and libraries ranging from the ingenious to the depressingly obscure.

Today, I'd like to talk to you about a pet project which is actually worth your time.

When I say worth your time, I mean a project which is financially, professionally, and personally rewarding in a way that pet software projects almost never are. Everybody likes money, of course; and this project, properly executed, is worth a small fortune. In fact, building and successfully running even a flawed and break-even version of the bot for several years is worth a decade of paychecks (for most people). But the money is (or should be) an afterthought, a nice-to-have.

The real reason I can suggest this project is as follows:

Building an online poker bot will give you a deep knowledge of two lucrative domains, programming and poker, and enough day-to-day working familiarity with those domains to position you, given some effort on your part, for the work of a professional software developer and/or poker player.

How does building a poker bot make you a better poker player? Because it forces you to confront, understand, and codify advanced poker strategy in a way that few of your opponents will have done. How does building a poker bot make you a better programmer? Because a bot involves a balanced coordination of techniques that span the breadth and depth of software development, from operating system nuts and bolts to abstract A.I., from user interfaces to multi-threading.

So to a certain degree, I'm the little devil on your left shoulder telling you to take the path less travelled by.

If I get my way, not only will you be building an online poker bot, you'll be playing poker seriously, for meaningful stakes - if you're not doing so already. You'll be programming seriously - if you're not doing so already. You'll leverage your programming skills to make you a better poker player. You'll leverage your poker-playing skills to make you a better programmer. And you'll leverage your natural (and healthy) desire for money to keep you interested in the process even when the going gets rough - and it will.

Looking at the source code for the bot, I'm reminded of another benefit: the creation of a generic code base which can be used to solve a great many poker problems other than online poker botting. If you structure this code base precisely, you don't have to put all your eggs in one basket. You can cautiously develop in the direction of an online poker bot, while giving yourself the option of branching off into other poker-related projects, many of which have significant commercial appeal (as the author of Poker Tracker could tell you).

Now, the last thing this series is, is a get-rich-quick scheme. But let's pull that copy of Theory of Poker off the shelf, blow the dust off the cover, and turn to the chapter on Implied Odds.

Implied odds are based on the possibility of winning money in later betting rounds over and above what is in the pot already. More precisely, your implied odds are the ratio of your total expected win when your cards hit to the present cost of calling a bet.

Implied odds dominate big-bet forms of poker such as No Limit and Pot Limit, and even affect play in Fixed Limit games, to a degree. But implied odds also apply to real-world scenarios of risk and reward. For example: let's say you're considering whether or not it's worth your time and effort to start building an online poker bot.

You can't simply calculate the likelihood of your being able to produce a winning poker bot, factor in the number of hours played, and so forth. You have to look at the hard-to-quantify value of the knowledge itself, and its effect on your career and life. You have to factor in the value of serendipity. In other words, you have to consider what you might call the implied odds of the situation.

• Will the bot make money? By winning money from other players? Via promotions? As a product?
• Will the skills I learn qualify me for a job I never could've gotten otherwise?
• Will I get addicted to poker and end up playing for a living, in Vegas, California, London?
• Will I meet a person or persons who will introduce me to other opportunities?
• Will I end up taking the bot's source code and selling it, or developing a full-fledged commercial application?
• Will I enter my bot in one of these "botting competitions" I've heard about?

Taking those factors into account, I think you'll come to the same conclusion I did: there's considerably more at stake here than making a few bucks running your bot on Poker Stars. The implied odds of building an online poker bot are huge, even if your bot never turns a profit.

If that hadn't been the case, I never would've sat down to write that first line of code...

Underground Poker Meets Best-Practices Software Development

This whole thing started, appropriately enough, in Dallas, Texas, where a booming IT corridor intersects a thriving underground poker scene - or what used to be a thriving underground poker scene.

All-night poker clubs in back-alley office lots and basements. Places with names like 6th Street, MMO's, Executive Game, DTD, DPO, Third Pair. Full-service underground casinos with professional dealers. ATMs. Widescreen TVs. Dollar-a-minute table massages. $60,000 bad beat jackpots. On-site kitchens stocked with every fried food known to man. And the #1 draw of most Dallas poker games: free beer.

Times have changed in Dallas and many of these clubs have closed their doors. For a few reasons:

• The uncapped 10% rake structure of many low-limit Dallas games, combined with a way-above-average dealer toke, plus the ubiquitous bad beat jackpot, sucking an additional $1 from every pot.
• The opening of multiple major (legal) cardrooms an hour north, across the Oklahoma state line.
• The recent return of robberies at gunpoint, unheard of in Dallas since the days of the road gamblers.
• Increased scrutiny of the Dallas underground poker scene in the local news.
• The reversal of local law enforcement's long-standing policy of turning a blind eye towards local poker games.

But once upon a time, you could play no-limit Hold'em from dusk 'til dawn, seven days a week, at fifty different clubs around town, for buy-ins ranging from $40 to $5000 or more. It reminds me of a movie every poker player either loves, or loves to hate: 

Playing poker in Dallas a few years ago, you felt like you were living a scene from Rounders, except that the situation in Dallas went way beyond anything Rounders ever portrayed. The sudden mass influx of new money into the game - thanks in part to Rounders, thanks in part to online poker, thanks in part to Moneymaker, Raymer, Varkonyi - meant that anyone with a spare $30,000 lying around could start up a club and recoup their investment in a matter of a few months. People were literally renting out warehouses and throwing weekly freeroll tournaments to lure new players into the cash games (which of course, are where the house makes all its money). These freerolls drew hundreds of players - not a big deal in Vegas, perhaps, but this is a state where running a raked poker game was and is a class A misdemeanor.

 § 47.04.  KEEPING A GAMBLING PLACE.  (a)  A person
commits an offense if he knowingly uses or permits another to use as
a gambling place any real estate, building, room, tent, vehicle,
boat, or other property whatsoever owned by him or under his
control, or rents or lets any such property with a view or
expectation that it be so used.
    (b)  It is an affirmative defense to prosecution under this
section that:   
    (1)  the gambling occurred in a private place;                                
    (2)  no person received any economic benefit other than
personal winnings;  and
    (3)  except for the advantage of skill or luck, the
risks of losing and the chances of winning were the same for all
participants.
    (c)  An offense under this section is a Class A misdemeanor.

That's just shy of a felony, and trust me when I say: the people running these games weren't exactly hardened criminals. They were guys with landscaping businesses, law practices, and programming jobs.

Anyway, there was so much new money (also known as dead money) on the tables, you didn't have to play particularly well, in order to win. All you had to do was be patient. The strategy went something like this:

• Wait for a monster hand.
• Get all your money in against players who don't notice or don't care that you haven't played a hand in four hours.
• Rinse and repeat.

It's the (exploitable) style of play known in poker circles as weak-tight and that, folks, is all it took to grind out a decent living in those underground Dallas poker games of yesteryear. This style, absent tilt, is still good enough to win in today's low-limit online games, though as we'll see, the tight-aggressive and loose-aggressive styles are really where it's at.

I mention all this because very early on it occurred to me: there's got to be a way to extract this ABC style of poker into a piece of software and then hook that software up to an online poker site. And if you're a programmer who plays online poker seriously, don't pretend you haven't thought the same thing.

After all, why else do we learn to program, if not to be able to solve real-world problems?

An Introduction to Advanced Poker A.I

Many of you have commented that the Input (getting information from the poker client) and Output (clicking buttons and otherwise manipulating the poker client) stages are low-hanging fruit: tedious, but not rocket science. The Processing stage, on the other hand - the cognitive element of the bot - is another story.

Morpheus: This is your last chance. After this, there is no turning back. You take the blue pill - the story ends, you wake up in your bed and believe whatever you want to believe. You take the red pill - you stay in Wonderland and I show you how deep the rabbit-hole goes.

Artificial intelligence is a huge and fascinating subject, even when applied to the (relatively) narrow domain of poker. A sample of recent papers from the University of Alberta's Computer Poker Research Group (CPRG) should give you a taste:

• Computer Poker
• Dealing with Imperfect Information in Poker
• Probabilities and Simulations in Poker
• Poker as a Testbed for Machine Intelligent Research
• Opponent Modeling in Poker (PDF)
• Using Selective Sampling Simulations in Poker
• Learning To Play Strong Poker
• Using Probabilistic Knowledge and Simulation To Play Poker
• Improved Opponent Modeling in Poker
• Opponent Modeling in Poker: Learning and Acting in a Hostile Environment
• The Challenge of Poker
• Approximating Game-Theoretic Optimal Strategies for Full-scale Poker
• Opponent Modeling and Search in Poker
• A Tool for the Direct Assessment of Poker Decisions
• Optimal Unbiased Estimators for Evaluating Agent Performance
• Algorithms and Assessment in Computer Poker
• Postgame Analysis of Poker Decisions
• Robust Strategies and Counter-Strategies: Building a Champion Level Computer Poker Player
• Using State Estimation for Dynamic Agent Modeling

Unfortunately, most cutting-edge poker A.I. is beyond the reach (given the average person's schedule) of the average programmer/poker-player:

I have lot of respect for the scholarly work that the CPRG and others have done, and I think it's the kind of work which will end up producing the very strongest poker intelligence. But I'd recommend you stay away from the advanced, graduate-level A.I. (unless you happen to be an enthusiast or expert in the field) in favor of learning basic and advanced poker strategy cold. Ultimately that will get you more bang for your mental buck.

Botting Rule #16: None of the techniques used by the CPRG or other organizations are necessary to produce a winning online poker bot.

Now, those are some fightin' words, so let me qualify them. The CPRG approach has always been a game-theoretic approach: finding optimal and/or exploitive strategies which attempt to model the game of poker "from scratch". That means a lot of abstract math and statistical modeling, a lot of terms like Nash equilibria and Best Response, and it means relatively little domain-specific knowledge of poker as it's played for real money in today's online cardrooms:

• How to incorporate (for example) Poker Tracker statistics into the bot's decision-making.
• How to take advantage of player loyalty incentives and promotions.
• How straightforward, ABC poker play wins in low-limit online games.
• Convenient rules of thumb (if you flop a set, you're almost never making a mistake to simply get all your money in on the flop) and how they can simplify processing.
• Table selection, table selection, table selection (the poker equivalent of location, location, location).

That's not to say that the people at the CPRG don't "know" poker, or that they couldn't build an online poker bot which would beat the pants off of any other online poker bot that had the temerity to compete against them. It's just that they have a different goal in mind: creating a generic, from-scratch poker intelligence which is capable of competing at a world-class level using pure A.I. That's miles and miles beyond what it takes to produce a profitable online poker bot. And while we're not going to rule out any work that the CPRG (or other organizations) have done, at the same time, let's be realistic. How much time can you devote to building an "optimal" online poker bot? Two years? Ten? Can you devote six solid months to learning (or re-learning) the math? No?

Then start small, by choosing one or more of the following shortcuts:

• Don't require that your bot actually win. Instead, play the online promotion game so that it makes money even if it's a slight overall loser.
• Force the bot to play short-stacked, and to leave the table when it's no longer short-stacked.
• Specialize the bot for a very specific game, such as "low-limit, heads up, No Limit Hold'em tourneys".
• Force the bot to play at a full table (or short-handed), and to leave the table when that changes.
• Allow the bot to rely on occasional human assistance, or even to notify (through a loud buzzer) a nearby human when it needs assistance.
• Use toy games (no-fold, half-street, static card, etc) to simplify analysis.

Getting the bot up and running on "training wheels" in this way is one of the things that makes this project feasible for the tinkerer or botting enthusiast.

Iterative, Incremental Development

We're not going to rely on the above shortcuts exclusively. We want to have our cake and eat it too: meaning, we want to produce a bot which is capable of taking these shortcuts, but which has enough cognitive flexibility to leverage more sophisticated approaches, as we code them, or as they become available.

Botting Rule #44: In early versions of the bot, rely on inventive shortcuts to cut into the complexity of the game, and schedule these shortcuts as milestones on the path to the development of a full-fledged bot.

It's not much fun to spend two years coding away in isolation only to discover upon completion that your bot is ridiculously weak, and overly complex given how poorly it plays. Delineate the work into bite-sized chunks which you can deploy and test immediately. 

You working programmers out there already know the score on this one. Formal software development methodologies from RUP to Agile have one thing in common: iterative, incremental development. Even if you've never worked with a formal development methodology, you know that every month or so, customers, clients, managers, coworkers - somebody of importance - wants to see what you've been up to. When it comes to building a poker bot, likely as not that person will be you. And that means it's more important, not less important, to set bite-sized, verifiable development goals. Otherwise it's easy to get lost in the labyrinth of Windows application development, poker domain modeling, and generic A.I. which comprise a bot. In fact, it's impossible not to get lost if we don't break the problem down into manageable pieces.

And one of the big tools we'll use to allow that (when it comes to the bot's A.I.) is an extremely simple form of what's called a multi-agent system, or MAS.

Pluggable Intelligence and the Multi-Agent System

This is a subject (A.I.) that we'll be returning to again and again; but for now, I want to lay out some basic architecture. This isn't the only approach, and I make zero representations as to whether it's the best, or even a particularly good, approach. But it is a working approach which has undergone the trial-by-fire of competitive, real-money play.

What we're looking for is a generic mechanism which allows any poker processing logic or intelligence whatsoever - from the simplest of "hot and cold" probability analyzers to the cutting-edge artificial brain of tomorrow - whether written by us or by a third party - to participate in the decision-making process for a given game situation.

Botting Rule #4: Pluggable intelligence is a key requirement for an evolving bot.

In order to do that, we have to design the bot's cognitive component such that it's capable of incorporating and sublimating any arbitrary poker A.I. In effect, we want to build a simple multi-agent system:

A multi-agent system (MAS) is a system composed of multiple interacting intelligent agents. Multi-agent systems can be used to solve problems which are difficult or impossible for an individual agent or monolithic system to solve. Examples of problems which are appropriate to multi-agent systems research include online trading, disaster response, and modeling social structures.

Speaking in more precise terms: consider creating a programmatic interface (I called mine DecisionModule) which allows any arbitrary component or application to participate in the voting process ultimately responsible for producing a discrete output...

• fold
• check
• bet
• raise
• call

...for a given game situation. Each "Decision Module" can be thought of as a distinct intelligent agent, operating independently of or in cooperation with all other such agents which comprise the bot. These agents might be source-level classes that you write yourself, or they might be third-party applications or DLLs for which you've written an adapter.

For example, let's say a very strong commercial poker playing application is released, and it has a programmatic interface. With a pluggable MAS, you can leverage that application's intelligence as a pluggable component inside your bot, running in parallel with whatever other components you've created or acquired. 

In practical terms, that means you can have a simple "hot and cold" probability analyzer running next to tomorrow's cutting-edge third-party "artificial brain"; from the bot's perspective, each such component is a black box. Table state is passed into a sequence of Decision Modules, each of which conforms to a uniform interface. Together, they produce the ultimate decision to call or fold, in the form of a distribution which specifies (for example):

• Raise 80% of the time
• Call 20% of the time

The specific modules shown above are just by way of example, and of course, you can design this out any way you please. The important thing is to promote the concept of the multi-agent system to a first-order architectural abstraction within the bot. This approach has a number of advantages, for the A.I. hobbyist in particular:

• Support for any existing or future poker A.I., winning system, or method.
• We avoid hard-coding our bot to a particular solution, which may prove to be naive.
• We're able to drop in new plugins as we develop them or find them on the open market.
• Any single plugin can contain any level of sophistication, from simple toy game analysis to the cutting-edge heuristics of tomorrow.
• We have the ability to assign weights and other valuations to components, add and remove components, and otherwise tweak the voting process.
• We decouple the cognitive elements of the bot from the sensory and motive elements.
• Analysis can be run either synchronously, in the foreground, or asynchronously, in the background (worker thread semantics).

Once that's in place, we can start leveraging specific algorithms in the pursuit of a plus-EV bot, starting with simple rules-based or simulation approaches and eventually graduating to some more sophisticated abstractions. Again, an iterative, incremental development path studded with bite-sized, achievable goals is the way to the money.

Modeling the Poker Domain

In the first part of this series, we decomposed the bot into the three classic stages of information processing: Input, Processing, and Output. I noted that each stage of processing should run independently of the other stages.

• The Input stage should know nothing about poker; it just tries to extract arbitrary data from a given application using a set of rules.
• The Processing stage should know nothing about screen-scraping, API hooking, DLL injection, or input simulation. It's job is to take an abstract model of a poker game and analyze it.
• The Output stage should know nothing except that it's been instructed to simulate certain mouse clicks at specific times.

That's a good architectural division which it's worth preserving. But in practice we'll find that there are other even more meaningful divisions. In particular, the division between "generic poker things" and "online poker things". 

Again and again, you'll find that online poker things want to derive from generic poker things. In the diagram above, we can see that StarsPokerTable inherits from OnlinePokerTable, which inherits from PokerTable. We can use true Liskov substitution here because a StarsPokerTable "IS A" OnlinePokerTable which "IS A" PokerTable. If a function expects a single PokerTable object as an argument, you can pass in an OnlinePokerTable or a further-derived StarsPokerTable instead without altering the correctness of the program.

That sort of polymorphism is something that will be hugely helpful, and we'll use it everywhere. The pattern is this: work with base classes to the extent possible, and let derived classes transparently implement specific functionality as needed. This same sort of relationship applies throughout the poker and online poker domains. In fact, the relationship is so strong that what you'll end up with (or at least, what I ended up with) is a domain which can be physically segregated into different physical components:

• generic poker "things" (living in, for example, PokerDomain.dll).
• generic online poker "things" (living in OnlinePokerDomain.dll, which depends on PokerDomain.dll).
• specific online poker "things" (living in StarsPokerDomain.dll or FullTiltPokerDomain.dll, which depend on OnlinePokerDomain.dll).

That, in turn, offers a number of programming benefits, not least of which is decomposition of the system into layers of bite-sized components (each consisting of at most a handful of classes) with strong physical boundaries. It's also a good way to structure the source code if you intend on creating a bot that can play at multiple venues.

N-Dimensional Opponent Modeling

As we slowly but surely work our way into a candidate A.I. for the poker bot, we'll need to start picking up certain techniques and adding them to our arsenal. One of these is the idea of representing a player's particular strategy or inclination as a point or region in N-dimensional (2D, 3D, 4D, etc) space.

Often in poker we categorize and describe our opponents based on two things:

• Selectivity. How selective an opponent is with regard to choosing hands to play. Selective opponents who play few hands are called "tight". Unselective opponents who play many hands are called "loose".
• Aggression. How much betting and/or raising (as opposed to calling and/or checking) an opponent engages in. Opponents who frequently bet and raise are aggressive; those who rarely bet and raise are passive.

We can express selectivity and aggression on a familiar, two-dimensional Cartesian coordinate system. Here, both selectivity and aggression are expressed as numbers between -1.0 and +1.0. At (1.0, 1.0) a player is maximally selective and aggressive. At (-1.0, -1.0) an opponent is minimally selective and aggressive. 

Note that the four quadrants of this coordinate system correspond to the four "player archetypes" identified in The Psychology of Poker and other books. This works well as a rough indication of an opponent's core playing style. But I like to take the above grid and add a third axis: deception.

Deceptive opponents frequently attempt to mislead their opponents with bluffs, slowplays, and other tactics. If a deceptive opponent bets, you can't be sure whether he's got a strong hand or nothing at all. On the other hand, many opponents are "honest". A big bet indicates a big hand.

That allows us to express a player's (approximate) strategy as a single point in three-dimensional space. For example, a player at [0.75, 0.75, -0.75] would be described as tight, aggressive, and straightforward (non-deceptive). We would expect this player to bluff and slowplay less often than the player at [0.75, 0.75, 1.0].

Taking it a step further, we can add additional dimensions to our graph. For example, let's add a fourth dimension: stack size. The player at [1.0, 1.0, 1.0, -1.0] is maximally aggressive, selective, deceptive when his stack size is towards the bottom of the range. The player at [0.0, 0.0, 0.0, 0.0] plays a non-descript (neither tight nor aggressive, nor loose nor passive, but somewhere in between) game of poker when his stack is roughly average. And the player at [-1.0, -1.0, -1.0, 1.0] is passive, loose, and non-deceptive when his stack size is large.

Once we get into the multi-dimensional approach, we start to see that a given player's strategy isn't really a point, although we may choose to express it that way for convenience; it's a collection of points which together comprise a strategy region in n-dimensional space, where each dimension is a particular strategy facet. This region can be mathematically described, tested for intersection with other regions, and so forth. Use your imagination.

Of course, selectivity and aggression are just two possible axes. We'll use these because they're common throughout the poker literature, but understand that the meaning and quantity of axes is up to you. In fact, it's worth your while to set up a generic mechanism by which you can capture and plot arbitrary groupings of statistics. This in fact is one of the purposes of the PokerFact abstraction (in the diagram at the top of this page) and its many associated classes.

But that subject is a post in itself. Stick it in your toolbox and we'll return to it at a later date.

Opponent Modeling with PokerTracker

PokerTracker is a commercially available online poker hand history analysis tool - that is, a tool which parses the human-readable hand histories emitted by a particular site, analyzes them, and displays various statistics about your play and the play of your opponents:

Every serious online poker player, without exception, either owns, or knows about, Poker Tracker. In fact, Poker Tracker statistics (VPIP, PTBB/100, etc) have been adopted by the poker playing community at large, and now serve as a sort of de facto terminology for advanced poker strategy. You can hardly read a major poker forum such as 2 + 2 or Pocket Fives without understanding Poker Tracker lingo:

baja15 is winning at 17.5/13.5. chargers in o7 is winning at 13.4/11. The specific numbers are not important; what is important is that both chargers and baja are almost always raising the pots they play. Seizing the initiative is crucial to winning at tight/aggressive poker. They have the table reacting to them. When they call, which is usually about 3.5% of the pots, they have a reason for doing so. If chargers or baja went back through all the hands they called a PFR with, they would be able to tell you why they did it every time. And they would never say “well, because KQ is playable in MP” or something vague like that. It would be “because I was getting the implied odds to play 33 for a set here—villain is a 10/2 nit with 100bb who only raises UTG with premium hands; I knew I’d get at least a big part of his stack if I hit my set.”

As useful as Poker Tracker is for online poker players, it's even more useful for online poker botters. In fact, Poker Tracker is the single most useful third-party tool in the poker bot-building arsenal and here's why:

• It serves up a relational database (with a well-documented schema) containing 90% of the opponent historical data you'll ever want or need.
• It uses a family of statistics which have the weight of countless hours of discussion behind them.
• It's a commercially supported application with one of the most active communities in poker.
• It's allowed, and even tacitly encouraged, by all major online poker venues.

Absent Poker Tracker and similar tools, you'd be forced to parse those hand history files yourself, and to develop a statistical framework around them yourself, and to test and verify the utility of that framework yourself. Poker Tracker does all of this for you. And while the Poker Tracker statistics aren't perfect, they've got the weight of mass usage behind them, and you'll be able to find countless discussions about the ins and outs of various statistics, what they mean, how to use them to adjust strategy, and so forth.

Poker Tracker allows you to start answering questions like:

• Is opponent X a loose or a tight player? Passive or aggressive? Winning or losing?
• Where are the holes in my (or my bot's) game? Am I playing too loosely in the small blind?
• Am I winning or losing? How many total hands have I played? What's my average win rate?

We'll take a look at some working source code before too long, but if you know your way around a database, leveraging Poker Tracker shouldn't be too much of an issue. Also, Poker Tracker isn't the only game in town. You can also take a look at Poker Academy Prospector and Hold'em Manager, just to name a couple.

The Advantage of Playing Short-Stacked

You'll notice that one of the Decision Modules in the above diagram is called ExpertShortStackedAnalyzer. That's my always-inventive name for a particular class which attempts to model a game of poker in which the stack sizes have been artificially shortened.

I've mentioned a couple times the advantages of creating a short-stacked bot but I don't think I've been clear enough about what those advantages are, and why they're so helpful. First of all, when I say a short-stacked bot, I really mean a short-stacked v1.0 bot. The idea should be to get something basic up and running first, before tackling deep-stacked no-limit Hold'em.

Now, let's define what a short-stacked bot is:

A short-stack bot is a bot which buys in for between 10 and 40 times the big blind, and which leaves the table when it wins much more than that.

Most online poker sites set the minimum allowed buy-in at 20 times the big blind (20xBB), or squarely in the "short-stacked sweet spot." In No Limit Hold'em: Theory and Practice, David Sklansky explains:

Playing a Short Stack

Since most of the money gets bet early in the hand, and betting all-in necessitates a showdown, short stacks should concentrate on playing premium hands and generally avoid bluffing. Play tightly preflop and try to get your money in when you likely have the edge against the hands your opponents might have.

Indeed, a totally algorithmic strategy of playing only premium preflop hands (say AA-77, AK and AQ) will beat almost any no limit game when you have a short stack of approximately twenty times the big blind. Ed suggests just such a strategy to beginners, in more detail, in his book, Getting Started in Hold'em.

That's not to say that such a strategy is optimal. For expert players, playing short routinely is a sure way to slash win rates. But it is remarkable that a robot playing a short stack can beat even relatively tough no limit games. Don't underestimate the intrinsic advantages of playing a short stack.

He goes on to give two reasons why the short-stack has an intrinsic advantage:

• It's often correct for deep stacks to play loosely against other deep stacks, but incorrect for them to do so against short stacks. So the short stack is often up against weaker-than-average hands versus his range.
• Once the short stack is all-in, opponents will keep betting, forcing players to fold. These folds benefit the short stack mathematically, the moreso because his additional risk is zero.

To which I'd add:

• The average player is 100% clueless about how to play, and how to play against, a short stack - whether in a cash game or tourney situation.

Another reason the short-stacked approach is so powerful is that commitment decisions become much easier as the stack sizes dwindle. In Professional No-Limit Hold'em, Matt Flynn, Sunny Mehta, and Ed Miller write:

When the effective stack sizes are 50BB [or less], you can easily create favorable SPRs for top pair and overpair hands with straightforward raising tactics.

If you're not familiar with the terminology, an SPR (stack-to-pot ratio) is simply the ratio of your stack (or technically, the smaller stack) to the total size of the pot on the flop. It's a numeric expression of risk-vs.-reward at the nexus of the hand which, among other things, tells you when you welcome aggression and are willing or even eager to get all of your money in. The beauty of playing short-stacked is that the mix hands which are correct to play when you're short-stacked always produce favorable SPRs for getting your money all-in.

From a programming perspective, that often (not always, but more often than you'd think) reduces the flop decision to a binary one:

• YES, I'm somehow going to get my money all-in here, probably with a straightforward bet or raise.
• NO, I'm not willing to commit. I fold.

That's huge, in terms of cutting into that complexity I was talking about. But it doesn't end there. Playing short-stacked desaturates the game in a number of ways that simplify your job as the programmer:

• A short-stacked bot rarely has to consider any actions beyond the flop. This has the effect of turning a 4-street game (preflop, flop, turn, river) into a 2-street game (preflop, flop).
• Preflop play, while subtle at times, is fairly easy to abstract into a set of rules. This is somewhat true even in deep-stacked play, but it's especially true in short-stacked play.
• Flop play, the crux of the hand, can be simplified through the application of SPRs and some basic commitment mathematics (as I explained above).
• Any mistakes or weaknesses in the bot can only be punished in small increments.
• As Sklansky notes, it's usually incorrect for a short stack to bluff. That means you don't have to implement game-theoretic bluffing, and you don't have to implement opportunistic bluffing. In fact, beyond an occasional continuation bet, which will put you all-in anyway, you don't have to think about bluffing at all.
• Short stacks don't care or worry about implied odds. Estimating implied odds is one of those fuzzy areas in which humans prosper and robots perish.
• Buying in short is appropriate when the other players at the table are better than you, which will often be the case with a home-grown bot.

So while I don't suggest that you rule out the idea of creating a bot that can play deep-stacked, I do think the first step on the road to that goal is creating a bot which is capable of playing short-stacked.

The Importance of Generic Code

If you've read this far, then you're presumably pretty interested in this subject. So a friendly word of advice.

One of the reasons many bots fail is because they're designed and developed in a vacuum. Best practices are abandoned in favor of the quick and dirty approach. Code that would never fly in a commercial production environment is created, tweaked, reinforced, cut-and-pasted. The result: a useless, hacked-together mess of spaghetti code, or what's known as a Big Ball of Mud.

In a project of this complexity, avoiding this sort of thing has to be a fundamental requirement from Day One. Even if you're not successful in building the bot, keep your options open. Maybe some of the code can be usefully applied to another project. At all costs, avoid burning a year or more of your time producing a glut of bad "poker botting" code. Keep it clean, keep it reusable.

One of the better ways to lift your code out of the trenches and increase its utility and longevity is to rely on a tried-and-true software development best practice:

Botting Rule #2: Code the domain, not the problem.

When we apply this rule to the domain of poker, what we get is a code base in which the classes and other abstractions have a strong correlation with meaningful nouns and verbs:

• poker player
• poker table
• online poker table
• poker game
• poker hand
• hand history
• poker action (check, raise, bet, fold, etc)

Whether you're building a bot, a commercial poker game, or a simulation tool, you're probably going to be interested in the concept of a poker table. That's a core abstraction which establishes an ordering amongst players. Similarly, you'll want some notion of what it means to be a poker player. That's another core abstraction. And so on, until you end up with a group of reusable classes similar to those shown in the diagram presented above.

Taken together, and with the addition of a motive intelligence, these classes comprise a poker bot. But they're also useful in creating a hand history analyzer. A game simulator. A multi-tabling helper tool. An online poker heads up display (HUD). And so forth. If you take the proper care to model things this way, you'll find that your code has useful applications quite aside from botting. That, in turn, further increases the utility of the work you're performing, boosting the "reward" half of the risk/reward equation we talked about in the Introduction.

Ignoring this bit of advice cost me six months (easily) of development time in the creation of the above bot.

Conclusion

We've covered a lot of material in this post, including:

• Why building a poker bot is an ideal pet project.
• A light introduction to advanced poker A.I.
• A candidate architecture for a generic "pluggable intelligence" poker bot A.I.
• How to use an iterative, incremental approach to apportion the bot into bite-sized pieces.
• Some working advice and best practices for modeling the poker domain.
• N-Dimensional Opponent Modeling.
• Poker Tracker, and its value to the player/botter.
• Why short-stacked bots are so tough to beat.
• Avoiding the poker bot "Ball of Mud" dilemma.

In Part Two of this series, I promised you some working code for interrogating the poker client for various game data. That code has been rolled into its own dedicated post, and in fact, this is how the series will be structured from here on out: long, rambling higher-level posts interspersed with short, targeted, source-code rich posts solving specific problems.

I'd also like to apologize for the delay in responding to some of your questions and comments. These are all excellent topics for discussion and I'd like to address them fully, in dedicated posts. But however we structure it, the poker botting series will continue in exhaustive detail. If you've enjoyed the first three parts of the series, and you haven't already done so, please subscribe in a news reader or over email. And as always, thank you for your time.

Until next time...

Tags: poker bot, A.I., online poker, poker

133 comment(s)

Yes! Leave a Comment. All fields are optional.