Welcome back! Last week I discussed the data model I’m using to store data, so this week I can finally get to how I actually scrape the data and place it in the data model. I’ll also visit standard numbers once more so we can look at the delta from last week!

Automation:

We're going to dabble in a little bit of code now. I'm using a language called Apex which is really a derivative of Java localized to Salesforce, the product I use as my data warehouse software. I'll include the snippets of code, and then an explanation afterwards. If you aren't a coder, hopefully the descriptions will give you a good idea about how these sorts of technologies work. If you are a coder you can ignore the easy stuff.

As discussed previously, I want to store event data, and I can get that event data from Wizards. The initial target is the following URL, which will give us back a list of events over the last 14 days: http://www.wizards.com/handlers/XMLListService.ashx?dir=mtgo&type=XMLFileInfo&start=14

When your browser requests a URL, you get back some text which can be rendered as a web page if it's HTML, or viewed as a text file, XML file, pdf, etc depending on some definitions in the response to your request. URL requests are not restricted to browsers however. If you make a URL GET request in code, you get back the same text/xml/html you'd get in your browser, but you have the option on how to interpret the response programmatically. Let me show you with the first bit of code:

string url = 'http://www.wizards.com/handlers/XMLListService.ashx?dir=mtgo&type=XMLFileInfo&start=14';
HttpRequest req = new HttpRequest();
req.setEndpoint(url);
req.setMethod('GET');
Http http = new Http();
HTTPResponse res = http.send(req);

JSONParser parser = JSON.createParser(res.getBody());

Stepping through it line by line, you can see that I define the URL, define a new HttpRequest that I call req, I set the URL target for req as the URL, I set the type of request as a "GET" request, and then I create an HTTPResponse called res as the result of sending my HttpRequest req. The end result, is that the body of the HttpResonse res will look something like this:

[{"__type":"XMLFileInfo:#Wotc.Web.DataDefinitions","Date":"10\/23","Hyperlink":"6118427","Name":"Legacy Daily"},{"__type":"XMLFileInfo:#Wotc.Web.DataDefinitions","Date":"10\/23","Hyperlink":"6118429","Name":"Modern Daily"},...,...]

The last line takes the body from res and uses a JSON method to parse it. Boiling that down a bit, it makes the format of the returned information a little easier to work with in code by creating a list of smaller elements. Now let's break down the data even further:

MTGO_Event__c e;
Map Event_Insert_Map = new Map();

In this code block, I define a few things. The first is a new record e belonging to the event table. The second is a Map called Event_Insert_Map, which is like a list of records, except that every record has a unique handle that allows me to pull it back out of the list easily. Next, we are going to use the data we got from the first block of code:

while (parser.nextToken() != null) {
if(string.valueOf(parser.getCurrentToken()) == 'START_OBJECT'){
e = new MTGO_Event__c();
} else if (string.valueOf(parser.getCurrentToken()) == 'VALUE_STRING' && parser.getCurrentName() == 'Date'){
e.Event_Date__c = date.parse(parser.getText()+'/'+string.valueOf(date.today().year()));
} else if (string.valueOf(parser.getCurrentToken()) == 'VALUE_STRING' && parser.getCurrentName() == 'Name'){
e.Event_Type__c = parser.getText();
} else if (string.valueOf(parser.getCurrentToken()) == 'VALUE_STRING' && parser.getCurrentName() == 'Hyperlink'){
e.Event_Number__c = parser.getText();
e.name = e.Event_Number__c;
} else if (string.valueOf(parser.getCurrentToken()) == 'END_OBJECT' && (e.Event_Type__c.contains('Daily') || e.Event_Type__c.contains('Premier')) ){
Event_Insert_Map.put(e.Event_Number__c,e);
}
}

The first line creates a loop that says, while there is still another item in the JSON parser list keep doing all the things below me. The next few lines are all very repetitive. They look at the piece of text from the JSON parser list, and they do different things depending on the label of that item. If the label says START_OBJECT, create a new record. If the label says Date, Name, or Hyperlink, set the corresponding field in the record e to that value. After these lines have been processed, we should have something that looks like this:

MTGO_Event__c e
e.Event_Date__c = 10/23
e.Event_Type__c = Legacy Daily
e.Event_Number__c = 6118427

And finally, when the END_OBJECT label appears, we place e into the Event_Insert_Map map. We then repeat the cycle until all the items in the list have been exhausted. The final step is an easy one, and looks like this:

insert Event_Insert_Map.values();

We take the values of the Event_Insert_Map (all the records, without their corresponding handles) and we insert them into the database. Now we have persisted all the events from the last 14 days! Since I do this every day, I also have a check where I make sure not to make any duplicates. I do this overlap in case there is an error, so I have some time to correct the problem!

Next week I'll increase the complexity a little bit by using the list of events to scrape all the individual decks and store them as well.

Speculation:

Now, enough of that technical nonsense! I'm going to report on the standard environment again this week, so we can look at the delta between last weeks article and now, 7 days later. I spent a ton of time working on the styling of the tables and I'm also going from 50 rows down to 25 rows on each of the tables. The end result should be a much more refined and easy to read look. If you prefer it this way let me know.

Mythics:

Rares:

My thoughts:

• Voice of Resurgence fell in number of copies played, and busted through it's 30 ticket floor. I'm definitely looking to buy some sets of this card, but I'm hoping for a floor under 25. Going to watch the prices over the next few days and make a decision if this card goes no lower.
• Nightveil Specter is putting up really strong numbers with a very low price for a rare of that quantity. Obviously the best time to buy has already past, but if you are late to the party there might still be room to go in. If you have copies, the strong performances should support a continued rise, so hold!
• Master of Waves and Thassa, God of the Sea are still dipping in price but have maintained their position in the field. The floor should be any day now, with redemption opening very soon!

Conclusion:

Feel free to point out any of your own conclusions in the comments, I love hear great ideas. Thanks for the feedback last week on formatting, I hope things are much easier to read now!