Hello again! I've had to slow down my contributions to this site to bi-weekly for the moment. I have a wedding in 60 days and it's amazing how many little things there are to take care of. Previously I walked through some code I was using to enumerate the events Wizards was posting. Today I'm going to walk you through how I scrape them. At the end of the article, I want to discuss Liliana and see.

Automation:

Today lets dabble in a little more code. I'm not going to go overly deep because I expect most readers of this site aren't developers, but I'm hoping it will be interesting to get a basic idea of how web scraping can work.

Previously we used the URL http://www.wizards.com/handlers/XMLListService.ashx?dir=mtgo&type=XMLFileInfo&start=14 to get a list of rows that look like this:

MTGO_Event__c e
e.Event_Date__c = 10/23
e.Event_Type__c = Legacy Daily
e.Event_Number__c = 6118427

Now we want to focus on scraping the actual event data, and we can do so using the event number field. Looking at the URL for a recent Legacy event, you can see that the event number is used as the unique identifier for the event in the URL:
http://www.wizards.com/Magic/digital/MagicOnlineTourn.aspx?x=mtg/digital/magiconline/tourn/6257942

To dynamically find all the event pages in code, I just need to create a URL variable with the event number dynamically added at the end like so:
string url = 'http://www.wizards.com/Magic/Digital/MagicOnlineTourn.aspx?x=mtg/digital/magiconline/tourn/'+e.Event_Number__c;

Once again, we are going to use this URL to make an HTTP GET request in code to get back the source code of the event detail page. This should look familiar:

HttpRequest req = new HttpRequest();
req.setEndpoint(url);
req.setMethod('GET');
Http http = new Http();
HTTPResponse res = http.send(req);
string resNoWhite = res.getBody().deleteWhitespace().substring(resNoWhite.indexOf('<ahref="#decklists">Decklists</a>'),resNoWhite.indexOf('<p><b>Standings</b></p>'));

The final line may look a little complicated. In a manner similar to the way Excel allows the chaining of formula commands I am taking the response body and performing some modifications. First I delete all the spaces in the source code, and then I use the substring command to grab the middle chunk between a start point and endpoint of my choosing. This allows me to filter out a lot of the cruft. I'm left with a variable string named resNoWhite filled with the source code for the event detail page. Here is an abridged look at what this looks like:

<liid="ctl00_ctl00_ContentPlaceHolder1_mainContent_decklistLi"><ahref="#decklists">Decklists</a></li>
<liid="ctl00_ctl00_ContentPlaceHolder1_mainContent_standingsLi"><ahref="#standings">Standings</a></li>
</ul>
<divclass='blurb'>
<divid="ctl00_ctl00_ContentPlaceHolder1_mainContent_ArchiveHeadingPanel"class="heading">
<spanid="ctl00_ctl00_ContentPlaceHolder1_mainContent_TournBlurb"></span>
</div>
</div>
<divid="content">
<br/>
<br/>
<aname='decklists'>&nbsp;</a><p><b>Decklists</b></p><root>
<divclass="deck">
<divclass="decktop">
<divclass="decktopmiddle">
<divstyle="float:left">
<divclass="main">
<heading>Boin(4-0)</heading>
</div>
<divclass="sub">LegacyDaily#6257942on11/08/2013</div>
</div>
<divclass="deckoptions"style="display:none;">
<atarget="_blank"href="/magic/samplehand.asp?x=mtg/digital/magiconline/tourn/6257942&amp;decknum=1">
<imgsrc="/magic/assets/decklist/handIcon.png"border="0"alt="Viewasamplehandofthisdeck"align="right"/>
</a>
</div>
<divclass="dekoptions">
<ahref="/magic/mtgdigitalmagiconlinetourn6257942x1.dek?x=mtg/digital/magiconline/tourn/6257942&amp;decknum=1">
<imgsrc="/magic/assets/decklist/dekdownload.png"border="0"alt="Downloada.dekfileforuseinMagicOnline"align="right"/>
</a>
</div>
<brclass="clear"/>
</div>
</div>
<divclass="maindeck">
<divclass="maindeckmiddle">
<divstyle="position:relative;">
<tableclass="cardgroup">
<tr>
<tdalign="center"colspan="2">
<pclass="decktitle">MainDeck</p>
<pclass="cardcount">60cards
</p>
</td>
<tdalign="center"valign="top"style="width:230px"/>
</tr>
<tr>
<tdvalign="top"width="185">2
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Badlands"onclick="autoCardWindow(this)"href="javascript:void()">Badlands</a><br/>3
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Bayou"onclick="autoCardWindow(this)"href="javascript:void()">Bayou</a><br/>4
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Bloodstained_Mire"onclick="autoCardWindow(this)"href="javascript:void()">BloodstainedMire</a><br/>1
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Snow-Covered_Forest"onclick="autoCardWindow(this)"href="javascript:void()">Snow-CoveredForest</a><br/>1
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Snow-Covered_Mountain"onclick="autoCardWindow(this)"href="javascript:void()">Snow-CoveredMountain</a><br/>1
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Snow-Covered_Swamp"onclick="autoCardWindow(this)"href="javascript:void()">Snow-CoveredSwamp</a><br/>1
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Taiga"onclick="autoCardWindow(this)"href="javascript:void()">Taiga</a><br/>4
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Verdant_Catacombs"onclick="autoCardWindow(this)"href="javascript:void()">VerdantCatacombs</a><br/>4
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Wasteland"onclick="autoCardWindow(this)"href="javascript:void()">Wasteland</a><br/>3
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Wooded_Foothills"onclick="autoCardWindow(this)"href="javascript:void()">WoodedFoothills</a><br/><hrsize="1"width="50%"align="left"class="decktotals"/><spanclass="decktotals">24lands</span><br/><br/>4
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Bloodbraid_Elf"onclick="autoCardWindow(this)"href="javascript:void()">BloodbraidElf</a><br/>4
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Dark_Confidant"onclick="autoCardWindow(this)"href="javascript:void()">DarkConfidant</a><br/>4
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Deathrite_Shaman"onclick="autoCardWindow(this)"href="javascript:void()">DeathriteShaman</a><br/>2
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Grim_Lavamancer"onclick="autoCardWindow(this)"href="javascript:void()">GrimLavamancer</a><br/>4
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Tarmogoyf"onclick="autoCardWindow(this)"href="javascript:void()">Tarmogoyf</a><br/><hrsize="1"width="50%"align="left"class="decktotals"/><spanclass="decktotals">18creatures</span><br/><br/></td>
<tdvalign="top"width="185">3
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Abrupt_Decay"onclick="autoCardWindow(this)"href="javascript:void()">AbruptDecay</a><br/>3
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Hymn_to_Tourach"onclick="autoCardWindow(this)"href="javascript:void()">HymntoTourach</a><br/>4
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Lightning_Bolt"onclick="autoCardWindow(this)"href="javascript:void()">LightningBolt</a><br/>4
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Liliana_of_the_Veil"onclick="autoCardWindow(this)"href="javascript:void()">LilianaoftheVeil</a><br/>4
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Thoughtseize"onclick="autoCardWindow(this)"href="javascript:void()">Thoughtseize</a><br/><hrsize="1"width="50%"align="left"class="decktotals"/><spanclass="decktotals">18otherspells</span><br/><br/><divclass="decktitle"style="padding-bottom:8px;"><b><i>Sideboard</i></b></div>2
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Engineered_Plague"onclick="autoCardWindow(this)"href="javascript:void()">EngineeredPlague</a><br/>2
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Kitchen_Finks"onclick="autoCardWindow(this)"href="javascript:void()">KitchenFinks</a><br/>4
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Leyline_of_the_Void"onclick="autoCardWindow(this)"href="javascript:void()">LeylineoftheVoid</a><br/>2
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Mindbreak_Trap"onclick="autoCardWindow(this)"href="javascript:void()">MindbreakTrap</a><br/>2
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Pyroblast"onclick="autoCardWindow(this)"href="javascript:void()">Pyroblast</a><br/>1
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Sylvan_Library"onclick="autoCardWindow(this)"href="javascript:void()">SylvanLibrary</a><br/>2
<aclass="nodec"onmouseover="ChangeBigCard(1,this)"keyName="name"keyValue="Umezawa[s_Jitte"onclick="autoCardWindow(this)"href="javascript:void()">Umezawa'sJitte</a><br/><hrsize="1"width="50%"align="left"class="decktotals"/><spanclass="decktotals">15sideboardcards</span><br/><br/></td>
<tdvalign="top">
<divstyle="height:220px;width:185px;">
<span/>
</div>
</td>
</tr>
</table>
<divclass="mtgbigcardsidedeck"id="bigcard_1">
<imgsrc="http://Gatherer.wizards.com/Handlers/Image.ashx?size=small&amp;multiverseid=10100000&amp;type=card"class="sidedeckcard"/>
</div>
</div>
<br/>
</div>
</div>
</div>
<divclass="deck">
<divclass="decktop">
<divclass="decktopmiddle">
<divstyle="float:left">
<divclass="main">
<heading>Pranayama(3-1)</heading>
</div>
<divclass="sub">LegacyDaily#6257942on11/08/2013</div>
</div>
<divclass="deckoptions"style="display:none;">
<atarget="_blank"href="/magic/samplehand.asp?x=mtg/digital/magiconline/tourn/6257942&amp;decknum=2">
<imgsrc="/magic/assets/decklist/handIcon.png"border="0"alt="Viewasamplehandofthisdeck"align="right"/>
</a>
</div>
<divclass="dekoptions">
<ahref="/magic/mtgdigitalmagiconlinetourn6257942x2.dek?x=mtg/digital/magiconline/tourn/6257942&amp;decknum=2">
<imgsrc="/magic/assets/decklist/dekdownload.png"border="0"alt="Downloada.dekfileforuseinMagicOnline"align="right"/>
</a>
</div>
<brclass="clear"/>
</div>
</div>
<divclass="maindeck">
<divclass="maindeckmiddle">
<divstyle="position:relative;">
<tableclass="cardgroup">
<tr>
<tdalign="center"colspan="2">
<pclass="decktitle">MainDeck</p>
<pclass="cardcount">60cards
</p>
</td>
<tdalign="center"valign="top"style="width:230px"/>
</tr>
<tr>
<tdvalign="top"width="185">3
<aclass="nodec"onmouseover="ChangeBigCard(2,this)"keyName="name"keyValue="Ancient_Tomb"onclick="autoCardWindow(this)"href="javascript:void()">AncientTomb</a><br/>2
ETC..... (Abridged)

Now we get to the magic. How do you take a long chunk of text like the above and extract the juicy bits that are obviously in there, like place, cardname, and quantity? I use one of my favorite things in the whole wide world: regular expressions! If you aren't familiar with regular expressions (regex for short), wikipedia defines them as: a sequence of characters that forms a search pattern, mainly for use in pattern matching with strings, or string matching. If you have ever seen one, they can be complicated and horrible to look at but can make the process of filtering through text very very simple!

Pattern Deck_Pattern;
if(ev.Event_Type__c.contains('Premier')){
Deck_Pattern = Pattern.compile('[a-zA-Z0-9._ ]+\\([12345678][\\w]+\\)');
} else if(ev.Event_Type__c.contains('Daily')){
Deck_Pattern = Pattern.compile('[a-zA-Z0-9._ ]+\\([43]-[10]\\)');
}
Matcher Deck_Match = Deck_Pattern.matcher(resNoWhite);

In the above code block I start the search process by creating a new regex pattern named Deck_Pattern. Then, depending on whether the event is Premier or Daily, I use a different pattern. For the Legacy event focused on in this article, the piece of text being searched for is Boin(4-0)

Focusing on the pattern, we can pick it apart piece by piece:

• <heading> = This is literal text, so it looks for an exact match.
• [a-zA-Z0-9._ ]+ = This block defines a character set that includes all the allowed characters in a user name. The + at the end says look for any character in this set repeated 1 or more times.
• \\([43]-[10]\\) = This block looks for the placing, (4-0) or (3-1). The slashes are escape characters since ( and ) have special meaning in regex, but I want to use them literally.
• </heading> = This again is literal, and looks for the exact match.

The final step of the code takes this pattern and uses it against the variable holding the source code body. By using this command repeatedly, I can hop to each new deck in the source code. In the next article I'll go a little deeper on the pattern matching for each individual deck (this will probably be the last article on this topic).

Speculation:

For the speculation side of things, I thought it might be fun to focus on Liliana of the Veil, a card everyone has been wondering about when to purchase! I've harvested 4,280 Modern decks and 1006 Legacy decks in the last 6 months, so there are quite a few numbers to look at!

Modern Stats:

Lifetime Winning Decks = 699
Percentage Of Winning Decks = 16.33%

Lifetime Winning Quantity = 2581
Average Quantity Per Deck = 3.69

Legacy Stats:

Lifetime Winning Decks = 192
Percentage Of Winning Decks = 19.08%

Lifetime Winning Quantity = 496
Average Quantity Per Deck = 2.58

Financial Stats:

This is a chart of the Supernova bot's buy prices over the past 2 weeks, with a 2 day simple moving average and 10 day simple moving average added for comparison.

Analysis:

Liliana of the Veil is a tough lady to figure out. She did not devalue as much as I had hoped with rotation, but I'm not sure this makes her a bad target. Being played in 16% of modern decks and 19% of legacy decks is a great sign for her continued desirability. The fact that she is from a recent, heavily opened set and still maintains the status as one of the most expensive mythics in the game gives me a lot of confidence in her future price trajectory. I was a big fan of Jace, the Mind Sculptor several years back for this exact reason and that card went back over 80 tickets after its rotation drop. I am usually afraid of locking up my capital in an expensive target, but there is also something to be said in the efficiency of this kind of speculation. I may make 300% on a junk rare that goes big, but it will take me a lot of time to get that 300% back out of my cards if I have 200 or more of them to sell. A price gain of 15 tickets on 20 copies could result in 300 tickets of profit for just a few minutes of work with the bots. As my wedding approaches the value I'm placing on my time has begun to skyrocket, so a move like this is right up my alley right now.

In regards to the price chart, I think the 10 day moving average does a really good job of showing when good times to buy are. Until the current price drops below the moving average, I'm probably going to stay out of Liliana of the Veil. I'm going to watch carefully over the next 2 weeks and hopefully pull the trigger as she gets closer to a sell price of 40 tickets. Generally this would be a buy price on Supernova of around 37-38 tickets.

Conclusion:

That's it for this week, thanks for reading! Next time I'll finish up my discussion on scraping decklist data.

On a completely separate note, I have a beta account for Hearthstone and an alpha account for Hex. It's sad to compare the state of these products to the state of the MTGO beta. Hex seems particularly targeted at the MTGO user base, as the game is very similar in many aspects. Unfortunately neither of these games have mobile applications just yet, but when they do they could pose serious competition to MTGO. Anyone else trying these games?