Playstation network profile

I decided to add game achievements to my blog in gamelog section. I mainly playing now on my PS3 and I have playstation network account. Unfortunately there is no official API for PSN network, because of that I decided to write a simple page scrapper for getting information from https://secure.us.playstation.com/logged-in/trophies/public-trophies/ page. The profile page heavily use javascript and Ajax by which I cannot use mechanize gem. For this type of task i will use PhantomJS which is a headless WebKit with scriptable JavaScript API. It don’t use a lot of ram and don’t need Xvfb to be runned.

Instalation

Installation is very simple. First we need to download PhantomJS binaries.

cd /usr/local/share
sudo wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.7-linux-x86_64.tar.bz2
--2014-08-20 19:27:40-- https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.7-linux-x86_64.tar.bz2 Translacja bitbucket.org (bitbucket.org)... 131.103.20.168, 131.103.20.167 Łączenie się z bitbucket.org (bitbucket.org)|131.103.20.168|:443... połączono. Żądanie HTTP wysłano, oczekiwanie na odpowiedź... 302 FOUND Lokalizacja: https://bbuseruploads.s3.amazonaws.com/ariya/phantomjs/downloads/phantomjs-1.9.7-linux-x86_64.tar.bz2?Signature=2U0DMU4Actbgkt9HMWlDCysmGBw%3D&Expires=1408556626&AWSAccessKeyId=0EMWEFSGA12Z1HF1TZ82 [podążanie] --2014-08-20 19:27:41-- https://bbuseruploads.s3.amazonaws.com/ariya/phantomjs/downloads/phantomjs-1.9.7-linux-x86_64.tar.bz2?Signature=2U0DMU4Actbgkt9HMWlDCysmGBw%3D&Expires=1408556626&AWSAccessKeyId=0EMWEFSGA12Z1HF1TZ82 Translacja bbuseruploads.s3.amazonaws.com (bbuseruploads.s3.amazonaws.com)... 54.231.15.9 Łączenie się z bbuseruploads.s3.amazonaws.com (bbuseruploads.s3.amazonaws.com)|54.231.15.9|:443... połączono. Żądanie HTTP wysłano, oczekiwanie na odpowiedź... 200 OK Długość: 13161396 (13M) [application/x-tar] Zapis do: `phantomjs-1.9.7-linux-x86_64.tar.bz2' 100%[===========================================================================================================================================================================>] 13.161.396 1,36MB/s w 14s 2014-08-20 19:27:56 (893 KB/s) - zapisano `phantomjs-1.9.7-linux-x86_64.tar.bz2' [13161396/13161396]

Now we extract and link them.

sudo tar xjf phantomjs-1.9.7-linux-x86_64.tar.bz2
sudo ln -s /usr/local/share/phantomjs-1.9.7-linux-x86_64/bin/phantomjs /usr/local/share/phantomjs
sudo ln -s /usr/local/share/phantomjs-1.9.7-linux-x86_64/bin/phantomjs /usr/local/bin/phantomjs
sudo ln -s /usr/local/share/phantomjs-1.9.7-linux-x86_64/bin/phantomjs /usr/bin/phantomjs

If everything goes well we should have working phantomjs command.

phantomjs --version
1.9.7

How it works

Scrapper will use two scripts psn.js which will setup PhantomJS environment and inject our second script psn_scrapper.js which will use jQuery to interact with the website. After running the output will be returned in JSON.

PhantomJS application

// require main classes page for phantomjs scrapper and system for access to arguments
var page     = require('webpage').create();
var system   = require('system');

// our response object that will be printed as json to STDOUT
var response = {
  logs: [],
  games: {}
}

// first we get first argument that is login
var login         = system.args[1];
response["login"] = login;

// any console put into response logs
page.onConsoleMessage = function(msg, source, line) {
  response["logs"].push(msg);
};

// for simple debugging
page.onAlert = function(msg) {
  console.log(msg);
};

// for communicating with webpage scrapper that will be injected into page. Each message contains action key
page.onCallback = function (data) {
  // if action is exit then exit phantomjs and output JSON response
  if (data["action"] == "exit") {
    console.log(JSON.stringify(response));
    phantom.exit();
  }

  // get fetched games by parser and put them into response
  if (data["action"] == "games") {
    response["games"] = data["object"];
  }

  // super simple logging
  if (data["action"] == "log") {
    response["logs"].push(data["object"]);
  }
};

// load psn network profile page where you enter username
page.open("https://secure.us.playstation.com/logged-in/trophies/public-trophies/", function(status) {
  if ( status === "success" ) {
    // if page load success, inject our scrapper script and run it with passed login in arguments
    page.injectJs("./psn_scrapper.js");

    page.evaluate(function(passed_login) {
      var scrapper = new TrophyGameScrapper(passed_login);
      scrapper.run();
    }, login);
  } else {
    phantom.exit();
  }
});

Main scrapper script

function TrophyGameScrapper(login) {
  this.login     = login;
}

// Send logs to our scrapper
TrophyGameScrapper.prototype.log = function(msg) {
  window.callPhantom({ action: "log", object: msg });
}

// Start scrapping by filing input with user login
TrophyGameScrapper.prototype.run = function() {
  this.log("Filing form with login: "+this.login)
  $('#trophiesId').val(this.login);
  $('#btn_publictrophy').click();
  this.wait_for_tile_grids();
}

// Wait unitil user profile is not loaded
TrophyGameScrapper.prototype.wait_for_tile_grids = function() {
  if ($("#trophyTrophyList .tile-grid-trophies").size() == 0) {
    this.log("Waiting for trophy list to load...");
    var self = this;
    setTimeout(function() { self.wait_for_tile_grids() }, 1000);
  } else {
    this.wait_for_more_games();
  }
}

// Each time more games button is visible click it. After it disappers scrap every game
TrophyGameScrapper.prototype.wait_for_more_games = function() {
  var self      = this;
  var more_link = $("#trophyTrophyList a.more-link:visible");

  if (more_link.size() == 0) {
    this.scrap_games();
  } else {
    this.log("Loading next page...");
    more_link.click();
    setTimeout(function() { self.wait_for_more_games(); }, 5000);
  }
}

TrophyGameScrapper.prototype.scrap_games = function() {
  this.log("Scraping...");
  var self = this;
  // find all game tiles
  var games = $("#trophyTrophyList .tile-grid-trophies .compete-tile").map(function(){
    var game_data      = {};
    var trophy_data    = {};
    var game_container = $(this);
    // get image, console and title information
    game_data["image"]      = "http:" + game_container.find(".trophy-image img").attr("src");
    game_data["console"]    = game_container.find(".titleLogoPS3").text();
    game_data["title"]      = game_container.find(".title").text();

    self.log("Found game: "+ game_data["title"]);

    // get game progress information
    trophy_data["progress"] = parseInt(game_container.find(".progress span").text().replace("%", ""));

    // and trophy count for bronze, silver, gold and platinum
    game_container.find(".trophies li").each(function() {
      var li = $(this);
      trophy_data[li.attr("class")] = parseInt(li.text());
    });

    game_data["trophies"]   = trophy_data;
    return game_data;
  }).toArray();

  // send games to phantomjs
  window.callPhantom({ action: "games", object: games });
  this.done();
}

TrophyGameScrapper.prototype.done = function() {
  window.callPhantom({ action: "exit" });
}

window.TrophyGameScrapper = TrophyGameScrapper;



Finally simple test if everything is working:

phantomjs psn.js macbury
{"logs":["Filing form with login: macbury","Waiting for trophy list to load...","Waiting for trophy list to load...","Waiting for trophy list to load...","Waiting for trophy list to load...","Loading next page...","Loading next page...","Scraping...","Found game: Mass Effect™ 3","Found game: Deus Ex: Human Revolution - Director's Cut","Found game: Proteus","Found game: GRID 2","Found game: Mass Effect™ 2","Found game: Crysis®3","Found game: Battlefield 4™ Trophies","Found game: MARVEL VS. CAPCOM 3","Found game: TEKKEN REVOLUTION","Found game: ACE COMBAT™ INFINITY","Found game: Mass Effect™","Found game: Lone Survivor","Found game: Far Cry® 3","Found game: Puppeteer™","Found game: The Last of Us™","Found game: FEZ","Found game: The Walking Dead","Found game: South Park™: The Stick of Truth™","Found game: XCOM: Enemy Unknown","Found game: Killzone 3","Found game: SOULCALIBUR Lost Swords","Found game: PAYDAY 2","Found game: Zone of the Enders: The 2nd Runner HD Edition","Found game: Sly Cooper: Thieves in Time™","Found game: Hotline Miami","Found game: BioShock Infinite","Found game: Uncharted 3: Drake's Deception™","Found game: Beyond: Two Souls™","Found game: LittleBigPlanet™ Karting","Found game: Brothers","Found game: SingStar™","Found game: PlayStation®Home"],"games":[{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR01814_00_00C673FFAB6E8B4E211A9DE573760C366509A30571/FEC5793C03CDF70FBDC39D24F153B77DC32E6C89.PNG","title":"Mass Effect™ 3","trophies":{"bronze":21,"gold":0,"platinum":0,"progress":27,"silver":2}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR05115_00_00E7EDB3F8D539CC02D6B96A3A00A89AB2ADCEF6A5/2064F520721F07D77B51694311AAB9C0B496D975.PNG","title":"Deus Ex: Human Revolution - Director's Cut","trophies":{"bronze":0,"gold":0,"platinum":0,"progress":0,"silver":0}},{"console":"PS3, PS VITA","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR05008_00_00EA4C91D38BEBF2150FB80A83DEA15B5FE8EFA060/B8236A34667C16286EB499EC8CF14AC7E273FAD0.PNG","title":"Proteus","trophies":{"bronze":0,"gold":0,"platinum":0,"progress":0,"silver":0}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR03961_00_0019BFF480D1A35BC20906563913CD0F0FBA8D380B/250A6522314441C0B4C935D7852B18D058C8D1C3.PNG","title":"GRID 2","trophies":{"bronze":1,"gold":0,"platinum":0,"progress":1,"silver":0}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR01580_00_006BED3D2E8564B72462EB11FE3042614297113D70/0DF2825C4C57EF0ECA6893BCAD22AA45070DEBB9.PNG","title":"Mass Effect™ 2","trophies":{"bronze":41,"gold":1,"platinum":0,"progress":74,"silver":4}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR03522_00_009C3F49A693A17BE657340AF1B9444266FA603F15/A6147872E0CABD699FC3CA349E8DC7E81B41FA87.PNG","title":"Crysis®3","trophies":{"bronze":0,"gold":0,"platinum":0,"progress":0,"silver":0}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR04412_00_00FABD2BE6C8EBFB6150A5446E1FD9351B8E9A9E77/146D0B429B3FC1799F97C68AB5F70FA69ABDDD41.PNG","title":"Battlefield 4™ Trophies","trophies":{"bronze":0,"gold":0,"platinum":0,"progress":0,"silver":0}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR01090_00_0030750786C6D32E42859035166173F8825A75334B/81B13E28D36FB8C181EB27879D70980EABD0E119.PNG","title":"MARVEL VS. CAPCOM 3","trophies":{"bronze":2,"gold":0,"platinum":0,"progress":2,"silver":0}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR04645_00_000777F82D284ECF66A7C1BA1DD4F02FC88CEB4AAB/1D67EFDBDC9D5180BA006150871E810BA1113BA3.PNG","title":"TEKKEN REVOLUTION","trophies":{"bronze":5,"gold":1,"platinum":0,"progress":71,"silver":2}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR04428_00_0018D0860A72F6701E04B2A8802A5CC8432C90AEB7/13A9F33BFF245A7DFF976A27D5A61FC08D81E938.PNG","title":"ACE COMBAT™ INFINITY","trophies":{"bronze":2,"gold":0,"platinum":0,"progress":9,"silver":0}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR04000_00_0067FBF0A5F1A43EAAE511145CC35DFDA4A53F1BC5/E6751CF420026E5B506646B869AB4419AB488069.PNG","title":"Mass Effect™","trophies":{"bronze":8,"gold":1,"platinum":0,"progress":40,"silver":7}},{"console":"PS3, PS VITA","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR04948_00_008036BD6B5353E0A91954362FDE8C503793167CC3/59FAAA55CED6ED1606CB0AF42EFBA5994344412A.PNG","title":"Lone Survivor","trophies":{"bronze":0,"gold":0,"platinum":0,"progress":0,"silver":0}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR03468_00_0066E2EC2FFDB9A69E7D4DAB927464E1758C8BB1D2/D8E6AF09350BC081DBFEEED20DD4AA95F7BEBB60.PNG","title":"Far Cry® 3","trophies":{"bronze":14,"gold":0,"platinum":0,"progress":25,"silver":2}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR01785_00_003F9F56B855B33D7EBF1F09C06ADF070ACCB44C0D/945EB6AD16AA97BD54F1AE4F2F7C0FAFEAFB7492.PNG","title":"Puppeteer™","trophies":{"bronze":6,"gold":0,"platinum":0,"progress":8,"silver":0}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR03073_00_00982ACA690865BF39A52624C65BDC9515DE4031EB/E0F3BAA831E39F5D3B948A165564E4C05C33D096.PNG","title":"The Last of Us™","trophies":{"bronze":2,"gold":0,"platinum":0,"progress":1,"silver":0}},{"console":"PS3, PS4, PS VITA","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR05526_00_0085EC163FE0BC2DDB5DB55195197043392BB23A50/E863E7A8B266C4EA61E14BF73245C14CF522176C.PNG","title":"FEZ","trophies":{"bronze":0,"gold":0,"platinum":0,"progress":0,"silver":0}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR03267_00_007FE16D7FFD3645788A29594335E3DDEBDC6E1BAD/3D9A794003EA37BC8EFE198E341A0CD36A4CD87E.PNG","title":"The Walking Dead","trophies":{"bronze":0,"gold":0,"platinum":0,"progress":0,"silver":0}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR03936_00_00433C5D11CE77C848E47D044DA5901063E4DEDC8A/818226016D44501829803DB253C380AFD7746B8A.PNG","title":"South Park™: The Stick of Truth™","trophies":{"bronze":19,"gold":0,"platinum":0,"progress":35,"silver":3}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR03580_00_00B99D09CFEC4BD556DCD9D36A6EE4A9A023F0DC94/C272E5FC7C1F30F7BFF84B6F2C483CB9A54EBB37.PNG","title":"XCOM: Enemy Unknown","trophies":{"bronze":1,"gold":0,"platinum":0,"progress":1,"silver":0}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR01569_00_0067015A3EE4202AB58C5806B0AC5F75782BF7A378/D491F79AF9DF6ED5DCA2F95AE7B48B2D9CC9A2B8.PNG","title":"Killzone 3","trophies":{"bronze":7,"gold":0,"platinum":0,"progress":6,"silver":0}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR04758_00_00DEBE109F017F5BE242B6057907338664F50A272B/95F63BF8D7B88A4260EA0B4F7DAD579D19A111C9.PNG","title":"SOULCALIBUR Lost Swords","trophies":{"bronze":0,"gold":0,"platinum":0,"progress":0,"silver":0}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR05046_00_0071CA099E3FA09BD7980AD3A2B2329226D59FD847/07114B3CEB47F6F8D6FD9390510A0F6DE923201B.PNG","title":"PAYDAY 2","trophies":{"bronze":1,"gold":0,"platinum":0,"progress":1,"silver":0}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR03197_00_00981EFD42A151989C71DA31F2F71F03AE2AEC3A98/63961F83317AD9B0DD297FAC9D4B4526B082019D.PNG","title":"Zone of the Enders: The 2nd Runner HD Edition","trophies":{"bronze":1,"gold":0,"platinum":0,"progress":13,"silver":4}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR03581_00_00166D98CDE19468AD3128794E762B1EE14849D570/442C12ABA76E9E3F219B6960F08353471B485F9A.PNG","title":"Sly Cooper: Thieves in Time™","trophies":{"bronze":0,"gold":0,"platinum":0,"progress":0,"silver":0}},{"console":"PS3, PS4, PS VITA","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR04533_00_00E51FAC067265BFA88E49688EEEF390EA705816AC/E73F8218212A3970AE3D304FE8095583E592932D.PNG","title":"Hotline Miami","trophies":{"bronze":0,"gold":0,"platinum":0,"progress":0,"silver":0}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR01478_00_00C06E65A82791FA660FF25F63B3FFE709006661F9/DEF5791FB8CC86CFAD8F32AD3FE2BEC8D3BADA93.PNG","title":"BioShock Infinite","trophies":{"bronze":3,"gold":0,"platinum":0,"progress":2,"silver":0}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR01216_00_0061B90D1E86DA292D0AD0B17AD2B7D83B459B65FE/992461104239EA3713FC4CE9B1F90EE09CBBEAD5.PNG","title":"Uncharted 3: Drake's Deception™","trophies":{"bronze":6,"gold":0,"platinum":0,"progress":4,"silver":0}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR03797_00_00CE033A46142B1FFFFE56DE24F7BFDAAF08AB250F/D2D0EFC4CF84A060692EBA7A117347E75DDB1257.PNG","title":"Beyond: Two Souls™","trophies":{"bronze":4,"gold":0,"platinum":0,"progress":6,"silver":0}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR02312_00_00AD8AE6C12FB21A4422F67B5647250E5502BB1A22/699FB54CBE02B5F32698BCBA235D2C66EB2CCEC9.PNG","title":"LittleBigPlanet™ Karting","trophies":{"bronze":4,"gold":0,"platinum":0,"progress":5,"silver":0}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR03959_00_00EB1934172162980AABD66B0EC81A144FAB48D4A6/0C4E97A1B1A1727725D40E49B9244F98A8B57CB9.PNG","title":"Brothers","trophies":{"bronze":0,"gold":0,"platinum":0,"progress":0,"silver":0}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR03204_00_00E2F6B60D4E88DC658B5FEC5184BA0E4EB7F04B6B/259D62AC555A746F8775F615B8B9B11BDE5902C5.PNG","title":"SingStar™","trophies":{"bronze":0,"gold":0,"platinum":0,"progress":0,"silver":0}},{"console":"PS3","image":"http://trophy01.np.community.playstation.net/trophy/np/NPWR00050_00_00F640216909F8C8C95B79AECEB84B54D4B2F05064/34573B2BAC22E236A60C6FEE61EE51EE4F49C1EE.PNG","title":"PlayStation®Home","trophies":{"bronze":2,"gold":0,"platinum":0,"progress":9,"silver":0}}],"login":"macbury"}

Integrating with ruby application

This step will be very easy. We need add phantomjs-gem to our Gemfile. This gem will automatically download phantomjs if is not installed in system. Next step is to run our script and capture STDOUT.

# Simple class for storing ps game info from json response
module Scraper
  class PSGame < Struct.new(:title, :image, :progress, :bronze, :silver, :gold, :platinum)
    def initialize(json={})
      self.title    = json["title"]
      self.image    = json["image"]

      trophies      = json["trophies"]
      self.progress = trophies["progress"]
      self.bronze   = trophies["bronze"]
      self.silver   = trophies["silver"]
      self.gold     = trophies["gold"]
      self.platinum = trophies["platinum"]
    end

    def sum_trophies
      self.bronze + self.silver + self.gold + self.platinum
    end
  end
end
require "phantomjs"
require "json"
require "scraper/game"

module Scraper
  class PSN
    attr_accessor :games

    def initialize(login)
      # Check where is phantomjs localted. If is not installed on the system, gem will download it and install in home directory
      log "Phantom js is located at: #{Phantomjs.path}"
      # run our script and process response
      response = JSON.parse(Phantomjs.run("--ssl-protocol=sslv3", "--web-security=false", Rails.root.join("lib/scraper/psn.js").to_s, login))
      response["logs"].each { |line| log(line) }

      self.games = response["games"].map { |raw_game| PSGame.new(raw_game) }
    end

    def log(msg)
      Rails.logger.info "[Scraper] #{msg}"
    end
  end
end

Comments

nedburglars on January 28, 2015 12:42

Everything seems to work ok, but i dont get any output when i enter phantomjs psn.js macbury

dan_benjamin on September 30, 2014 10:26

Awesome. Will try it out when I build my next scraper. Currently using Mechanize and hacking my way to extract cookies and csrf tokens.