{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Steam Data Collection\n", "\n", "*This forms part of a larger series of posts for my [blog](http://nik-davis.github.io) on downloading, processing and analysing data from the Steam store. [See all posts here](http://nik-davis.github.io/tag/steam).*" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "application/json": { "Software versions": [ { "module": "Python", "version": "3.7.3 64bit [MSC v.1900 64 bit (AMD64)]" }, { "module": "IPython", "version": "7.5.0" }, { "module": "OS", "version": "Windows 10 10.0.17763 SP0" }, { "module": "numpy", "version": "1.16.3" }, { "module": "pandas", "version": "0.24.2" } ] }, "text/html": [ "
SoftwareVersion
Python3.7.3 64bit [MSC v.1900 64 bit (AMD64)]
IPython7.5.0
OSWindows 10 10.0.17763 SP0
numpy1.16.3
pandas0.24.2
Thu May 30 12:21:17 2019 GMT Summer Time
" ], "text/latex": [ "\\begin{tabular}{|l|l|}\\hline\n", "{\\bf Software} & {\\bf Version} \\\\ \\hline\\hline\n", "Python & 3.7.3 64bit [MSC v.1900 64 bit (AMD64)] \\\\ \\hline\n", "IPython & 7.5.0 \\\\ \\hline\n", "OS & Windows 10 10.0.17763 SP0 \\\\ \\hline\n", "numpy & 1.16.3 \\\\ \\hline\n", "pandas & 0.24.2 \\\\ \\hline\n", "\\hline \\multicolumn{2}{|l|}{Thu May 30 12:21:17 2019 GMT Summer Time} \\\\ \\hline\n", "\\end{tabular}\n" ], "text/plain": [ "Software versions\n", "Python 3.7.3 64bit [MSC v.1900 64 bit (AMD64)]\n", "IPython 7.5.0\n", "OS Windows 10 10.0.17763 SP0\n", "numpy 1.16.3\n", "pandas 0.24.2\n", "Thu May 30 12:21:17 2019 GMT Summer Time" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# view software version information\n", "\n", "# http://raw.github.com/jrjohansson/version_information/master/version_information.py\n", "%load_ext version_information\n", "# %reload_ext version_information\n", "\n", "%version_information numpy, pandas" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![steam_logo](https://nik-davis.github.io/images/steam_logo_white.jpg)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The first time I used Steam was with an account I can't remember the name of, playing a game which could well have been [Dark Messiah of Might & Magic](https://store.steampowered.com/app/2100/Dark_Messiah_of_Might__Magic/), on a computer that would easily be blown away by the processing power of my phone today.\n", "\n", "It was sometime in 2006, when every game I bought came on a disk in a box that now gathers dust somewhere in my parents' house. Surprisingly my current PC has a disk drive, though I can't remember the last time used it.\n", "\n", "In order to play the multiplayer component of the game, annoyingly I had to install a piece of third-party software I had never heard of called Steam, creating the account I have long since lost in the process. Here's what that software probably looked like at the time:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "

Source: pcgevan via PC Gamer

" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Fast-forward a little over 10 years and the Steam Store is huge, ubiquitous as the home of PC gaming and distribution. Whilst physical copies still just about feel at home on consoles, the PC market has [long since moved digital](https://www.pcgamer.com/uk/analyst-says-digital-sales-made-up-92-percent-of-pc-game-market-in-2013/). In case you are not familiar, Steam is a digital store for purchasing, downloading and playing video games. It hosts a variety of community features, allows pushing game updates to users automatically, and gathers news stories relevant to each title. It's a bit like [Google's Play Store](https://play.google.com) or [Apple's App Store](https://www.apple.com/uk/ios/app-store/) for phones.\n", "\n", "A large part of Steam's success as a platform is due to its use of frequent sales, convenience as a unified digital game library, and the aforementioned shift to digital over physical. Whilst other platforms are emerging and gaining traction, there is likely no better resource for examining gaming over the last decade. With that in mind, if we can construct a dataset from Steam's data, we will have access to a wealth of information about [nearly 30,000 games](https://store.steampowered.com/about/) released since 2003, when Steam first launched." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Project Goals\n", "\n", "\n", "\n", "The motivation for this project is to download, process and analyse a data set of Steam apps (games) from the Steam store, and gain insights into what makes a game more successful in terms of sales, play-time and ratings. We will imagine that we have been approached by a company hoping to develop and release a new title, using the findings we provide them to inform decisions about how best to manage their budget and hopefully increase the success of their next release.\n", "\n", "The first step will be tackling data collection - the actual retrieval of data from Steam's servers and databases. In the future we'll look at cleaning the data, transforming it into a more useful state, then on to data exploration and analysis. Finally we'll summarise our findings in a non-technical report which would be sent to the fictional company in question.\n", "\n", "\n", "\n", "At the end of the data collection and cleaning stages, we'd like to end up with a table or database like this:\n", "\n", "name | id | information | owners | price | rating\n", "--- | --- | --- | --- | --- | ---\n", "awesome game | 100 | genres, descriptors, variables | 100,000 | £9.99 | 9/10\n", "generic shooter 4 | 200 | definitely the best shooter | 50,000 | £39.99 | 6/10\n", "... | ... | ... | ... | ... | ...\n", "\n", "We can then interrogate the data, and investigate whether particular attributes tend to result in more successful games. Metrics like ownership and ratings should help define the success of a title.\n", "\n", "## Data Acquisition\n", "\n", "There are a number of ways to get this information. Obviously we could search the web (and especially [kaggle](https://www.kaggle.com/datasets)) for existing datasets, however to avoid letting someone else get away with all that hard work (and mainly for the purposes of learning) we'll be acquiring all the data ourselves from scratch.\n", "\n", "Often when generating data the best place to start is to check for APIs. Fortunately Valve (the company behind Steam) make one available at https://partner.steamgames.com/. An [API](https://en.wikipedia.org/wiki/Application_programming_interface) such as this allows anyone to interface with data on a website in a controlled way, usually providing a host of useful features to the end-user. Typically an API is a great way for developers to allow access to databases and information on a server. Unfortunately this documentation doesn't include all access points, but others have documented this for us. [This](https://wiki.teamfortress.com/wiki/User:RJackson/StorefrontAPI) documentation of the StorefrontAPI will be particularly useful.\n", "\n", "We'll be able to get good information about the details of each game from the Steam API, however we're still missing information about popularity and sales. Luckily we can easily get this data from another website, SteamSpy.\n", "\n", "[SteamSpy](https://steamspy.com/about) is a Steam stats-gathering service and crucially has data easily available through its own API (documentation [here](https://steamspy.com/api.php)). It provides a number of useful metrics including an estimation for total owners of each game.\n", "\n", "We'll be retrieving data from both APIs and combining them to form our dataset. For the purposes of this project, we'll be performing as little data cleaning as possible at this stage, providing 'dirty' data for data cleaning, the next step in this project.\n", "\n", "## Section outline:\n", "\n", "- Create an app list from SteamSpy API using 'all' request\n", "- Retrieve individual app data from Steam API, by iterating through app list\n", "- Retrieve individual app data from SteamSpy API, by iterating through app list\n", "- Export app list, Steam data and SteamSpy data to csv files\n", "\n", "## API references:\n", "\n", "- https://partner.steamgames.com/doc/webapi\n", "- https://wiki.teamfortress.com/wiki/User:RJackson/StorefrontAPI\n", "- https://steamapi.xpaw.me/#\n", "- https://steamspy.com/api.php\n", "\n", "\n", "## Import Libraries\n", "\n", "We begin by importing the libraries we will be using. We start with [standard library imports](https://docs.python.org/3/library/), or those available by default in Python, then import the third-party packages. We'll be using [requests](https://2.python-requests.org/en/master/) to handle interacting with the APIs, then the popular [pandas](http://pandas.pydata.org/pandas-docs/stable/) and [numpy](https://docs.scipy.org/doc/numpy/index.html) libraries for handling the downloaded data." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "# standard library imports\n", "import csv\n", "import datetime as dt\n", "import json\n", "import os\n", "import statistics\n", "import time\n", "\n", "# third-party imports\n", "import numpy as np\n", "import pandas as pd\n", "import requests\n", "\n", "# customisations - ensure tables show all columns\n", "pd.set_option(\"max_columns\", 100)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Next, we define a general, all-purpose function to process get requests from an API, supplied through a URL parameter. A dictionary of parameters can be supplied which is passed into the get request automatically, depending on the requirements of the API.\n", "\n", "Rather than simply returning the response, we handle a couple of scenarios to help automation. Occasionally we encounter an SSL Error, in which case we simply wait a few seconds then try again (by recursively calling the function). When this happens, and generally throughout this project, we provide quite verbose feedback to show when these errors are encountered and how they are handled.\n", "\n", "Sometimes there is no response when a request is made (returns None). This usually happens when too many requests are made in a short period of time, and the polling limit has been reached. We try to avoid this by pausing briefly between requests, as we'll see later, but in case we breach the polling limit we wait 10 seconds then try again.\n", "\n", "Handling these errors in this way ensures that our function almost always returns the desired response, which we return in json format to make processing easier." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "def get_request(url, parameters=None):\n", " \"\"\"Return json-formatted response of a get request using optional parameters.\n", " \n", " Parameters\n", " ----------\n", " url : string\n", " parameters : {'parameter': 'value'}\n", " parameters to pass as part of get request\n", " \n", " Returns\n", " -------\n", " json_data\n", " json-formatted response (dict-like)\n", " \"\"\"\n", " try:\n", " response = requests.get(url=url, params=parameters)\n", " except SSLError as s:\n", " print('SSL Error:', s)\n", " \n", " for i in range(5, 0, -1):\n", " print('\\rWaiting... ({})'.format(i), end='')\n", " time.sleep(1)\n", " print('\\rRetrying.' + ' '*10)\n", " \n", " # recusively try again\n", " return get_request(url, parameters)\n", " \n", " if response:\n", " return response.json()\n", " else:\n", " # response is none usually means too many requests. Wait and try again \n", " print('No response, waiting 10 seconds...')\n", " time.sleep(10)\n", " print('Retrying.')\n", " return get_request(url, parameters)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Generate List of App IDs\n", "\n", "Every app on the steam store has a unique app ID. Whilst different apps can have the same name, they can't have the same ID. This will be very useful to us for identifying apps and eventually merging our tables of data.\n", "\n", "Before we get to that, we need to generate a list of app ids which we can use to build our data sets. It's possible to generate one from the Steam API, however this has over 70,000 entries, many of which are demos and videos with no way to tell them apart. Instead, SteamSpy provides an 'all' request, supplying some information about the apps they track. It doesn't supply all information about each app, so we still need to request this information individually, but it provides a good starting point.\n", "\n", "Because many of the return fields are strings containing commas and other punctuation, it is easiest to read the response into a pandas dataframe, and export the required appid and name fields to a csv. We could keep only the appid column as a list or pandas series, but it may be useful to keep the app name at this stage." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
appidname
010Counter-Strike
120Team Fortress Classic
230Day of Defeat
340Deathmatch Classic
450Half-Life: Opposing Force
\n", "
" ], "text/plain": [ " appid name\n", "0 10 Counter-Strike\n", "1 20 Team Fortress Classic\n", "2 30 Day of Defeat\n", "3 40 Deathmatch Classic\n", "4 50 Half-Life: Opposing Force" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "url = \"https://steamspy.com/api.php\"\n", "parameters = {\"request\": \"all\"}\n", "\n", "# request 'all' from steam spy and parse into dataframe\n", "json_data = get_request(url, parameters=parameters)\n", "steam_spy_all = pd.DataFrame.from_dict(json_data, orient='index')\n", "\n", "# generate sorted app_list from steamspy data\n", "app_list = steam_spy_all[['appid', 'name']].sort_values('appid').reset_index(drop=True)\n", "\n", "# export disabled to keep consistency across download sessions\n", "# app_list.to_csv('../data/download/app_list.csv', index=False)\n", "\n", "# instead read from stored csv\n", "app_list = pd.read_csv('../data/download/app_list.csv')\n", "\n", "# display first few rows\n", "app_list.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Define Download Logic\n", "\n", "Now we have the `app_list` dataframe, we can iterate over the app IDs and request individual app data from the servers. Here we set out our logic to retrieve and process this information, then finally store the data as a csv file.\n", "\n", "Because it takes a long time to retrieve the data, it would be dangerous to attempt it all in one go as any errors or connection time-outs could cause the loss of all our data. For this reason we define a function to download and process the requests in batches, appending each batch to an external file and keeping track of the highest index written in a separate file.\n", "\n", "This not only provides security, allowing us to easily restart the process if an error is encountered, but also means we can complete the download across multiple sessions.\n", "\n", "Again, we provide verbose output for rows exported, batches complete, time taken and estimated time remaining." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "scrolled": false }, "outputs": [], "source": [ "def get_app_data(start, stop, parser, pause):\n", " \"\"\"Return list of app data generated from parser.\n", " \n", " parser : function to handle request\n", " \"\"\"\n", " app_data = []\n", " \n", " # iterate through each row of app_list, confined by start and stop\n", " for index, row in app_list[start:stop].iterrows():\n", " print('Current index: {}'.format(index), end='\\r')\n", " \n", " appid = row['appid']\n", " name = row['name']\n", "\n", " # retrive app data for a row, handled by supplied parser, and append to list\n", " data = parser(appid, name)\n", " app_data.append(data)\n", "\n", " time.sleep(pause) # prevent overloading api with requests\n", " \n", " return app_data\n", "\n", "\n", "def process_batches(parser, app_list, download_path, data_filename, index_filename,\n", " columns, begin=0, end=-1, batchsize=100, pause=1):\n", " \"\"\"Process app data in batches, writing directly to file.\n", " \n", " parser : custom function to format request\n", " app_list : dataframe of appid and name\n", " download_path : path to store data\n", " data_filename : filename to save app data\n", " index_filename : filename to store highest index written\n", " columns : column names for file\n", " \n", " Keyword arguments:\n", " \n", " begin : starting index (get from index_filename, default 0)\n", " end : index to finish (defaults to end of app_list)\n", " batchsize : number of apps to write in each batch (default 100)\n", " pause : time to wait after each api request (defualt 1)\n", " \n", " returns: none\n", " \"\"\"\n", " print('Starting at index {}:\\n'.format(begin))\n", " \n", " # by default, process all apps in app_list\n", " if end == -1:\n", " end = len(app_list) + 1\n", " \n", " # generate array of batch begin and end points\n", " batches = np.arange(begin, end, batchsize)\n", " batches = np.append(batches, end)\n", " \n", " apps_written = 0\n", " batch_times = []\n", " \n", " for i in range(len(batches) - 1):\n", " start_time = time.time()\n", " \n", " start = batches[i]\n", " stop = batches[i+1]\n", " \n", " app_data = get_app_data(start, stop, parser, pause)\n", " \n", " rel_path = os.path.join(download_path, data_filename)\n", " \n", " # writing app data to file\n", " with open(rel_path, 'a', newline='', encoding='utf-8') as f:\n", " writer = csv.DictWriter(f, fieldnames=columns, extrasaction='ignore')\n", " \n", " for j in range(3,0,-1):\n", " print(\"\\rAbout to write data, don't stop script! ({})\".format(j), end='')\n", " time.sleep(0.5)\n", " \n", " writer.writerows(app_data)\n", " print('\\rExported lines {}-{} to {}.'.format(start, stop-1, data_filename), end=' ')\n", " \n", " apps_written += len(app_data)\n", " \n", " idx_path = os.path.join(download_path, index_filename)\n", " \n", " # writing last index to file\n", " with open(idx_path, 'w') as f:\n", " index = stop\n", " print(index, file=f)\n", " \n", " # logging time taken\n", " end_time = time.time()\n", " time_taken = end_time - start_time\n", " \n", " batch_times.append(time_taken)\n", " mean_time = statistics.mean(batch_times)\n", " \n", " est_remaining = (len(batches) - i - 2) * mean_time\n", " \n", " remaining_td = dt.timedelta(seconds=round(est_remaining))\n", " time_td = dt.timedelta(seconds=round(time_taken))\n", " mean_td = dt.timedelta(seconds=round(mean_time))\n", " \n", " print('Batch {} time: {} (avg: {}, remaining: {})'.format(i, time_td, mean_td, remaining_td))\n", " \n", " print('\\nProcessing batches complete. {} apps written'.format(apps_written))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Next we define some functions to handle and prepare the external files.\n", "\n", "We use `reset_index` for testing and demonstration, allowing us to easily reset the index in the stored file to 0, effectively restarting the entire download process.\n", "\n", "We define `get_index` to retrieve the index from file, maintaining persistence across sessions. Every time a batch of information (app data) is written to file, we write the highest index within `app_data` that was retrieved. As stated, this is partially for security, ensuring that if there is an error during the download we can read the index from file and continue from the end of the last successful batch. Keeping track of the index also allows us to pause the download, continuing at a later time.\n", "\n", "Finally, the `prepare_data_file` function readies the csv for storing the data. If the index we retrieved is 0, it means we are either starting for the first time or starting over. In either case, we want a blank csv file with only the header row to begin writing to, se we wipe the file (by opening in write mode) and write the header. Conversely, if the index is anything other than 0, it means we already have downloaded information, and can leave the csv file alone." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "def reset_index(download_path, index_filename):\n", " \"\"\"Reset index in file to 0.\"\"\"\n", " rel_path = os.path.join(download_path, index_filename)\n", " \n", " with open(rel_path, 'w') as f:\n", " print(0, file=f)\n", " \n", "\n", "def get_index(download_path, index_filename):\n", " \"\"\"Retrieve index from file, returning 0 if file not found.\"\"\"\n", " try:\n", " rel_path = os.path.join(download_path, index_filename)\n", "\n", " with open(rel_path, 'r') as f:\n", " index = int(f.readline())\n", " \n", " except FileNotFoundError:\n", " index = 0\n", " \n", " return index\n", "\n", "\n", "def prepare_data_file(download_path, filename, index, columns):\n", " \"\"\"Create file and write headers if index is 0.\"\"\"\n", " if index == 0:\n", " rel_path = os.path.join(download_path, filename)\n", "\n", " with open(rel_path, 'w', newline='') as f:\n", " writer = csv.DictWriter(f, fieldnames=columns)\n", " writer.writeheader()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Download Steam Data\n", "\n", "Now we are ready to start downloading data and writing to file. We define our logic particular to handling the steam API - in fact if no data is returned we return just the name and appid - then begin setting some parameters. We define the files we will write our data and index to, and the columns for the csv file. The API doesn't return every column for every app, so it is best to explicitly set these.\n", "\n", "Next we run our functions to set up the files, and make a call to `process_batches` to begin the process. Some additional parameters have been added for demonstration, to constrain the download to just a few rows and smaller batches. Removing these would allow the entire download process to be repeated." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Starting at index 0:\n", "\n", "Exported lines 0-4 to steam_app_data.csv. Batch 0 time: 0:00:10 (avg: 0:00:10, remaining: 0:00:10)\n", "Exported lines 5-9 to steam_app_data.csv. Batch 1 time: 0:00:10 (avg: 0:00:10, remaining: 0:00:00)\n", "\n", "Processing batches complete. 10 apps written\n" ] } ], "source": [ "def parse_steam_request(appid, name):\n", " \"\"\"Unique parser to handle data from Steam Store API.\n", " \n", " Returns : json formatted data (dict-like)\n", " \"\"\"\n", " url = \"http://store.steampowered.com/api/appdetails/\"\n", " parameters = {\"appids\": appid}\n", " \n", " json_data = get_request(url, parameters=parameters)\n", " json_app_data = json_data[str(appid)]\n", " \n", " if json_app_data['success']:\n", " data = json_app_data['data']\n", " else:\n", " data = {'name': name, 'steam_appid': appid}\n", " \n", " return data\n", "\n", "\n", "# Set file parameters\n", "download_path = '../data/download'\n", "steam_app_data = 'steam_app_data.csv'\n", "steam_index = 'steam_index.txt'\n", "\n", "steam_columns = [\n", " 'type', 'name', 'steam_appid', 'required_age', 'is_free', 'controller_support',\n", " 'dlc', 'detailed_description', 'about_the_game', 'short_description', 'fullgame',\n", " 'supported_languages', 'header_image', 'website', 'pc_requirements', 'mac_requirements',\n", " 'linux_requirements', 'legal_notice', 'drm_notice', 'ext_user_account_notice',\n", " 'developers', 'publishers', 'demos', 'price_overview', 'packages', 'package_groups',\n", " 'platforms', 'metacritic', 'reviews', 'categories', 'genres', 'screenshots',\n", " 'movies', 'recommendations', 'achievements', 'release_date', 'support_info',\n", " 'background', 'content_descriptors'\n", "]\n", "\n", "# Overwrites last index for demonstration (would usually store highest index so can continue across sessions)\n", "reset_index(download_path, steam_index)\n", "\n", "# Retrieve last index downloaded from file\n", "index = get_index(download_path, steam_index)\n", "\n", "# Wipe or create data file and write headers if index is 0\n", "prepare_data_file(download_path, steam_app_data, index, steam_columns)\n", "\n", "# Set end and chunksize for demonstration - remove to run through entire app list\n", "process_batches(\n", " parser=parse_steam_request,\n", " app_list=app_list,\n", " download_path=download_path,\n", " data_filename=steam_app_data,\n", " index_filename=steam_index,\n", " columns=steam_columns,\n", " begin=index,\n", " end=10,\n", " batchsize=5\n", ")" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "scrolled": false }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
typenamesteam_appidrequired_ageis_freecontroller_supportdlcdetailed_descriptionabout_the_gameshort_descriptionfullgamesupported_languagesheader_imagewebsitepc_requirementsmac_requirementslinux_requirementslegal_noticedrm_noticeext_user_account_noticedeveloperspublishersdemosprice_overviewpackagespackage_groupsplatformsmetacriticreviewscategoriesgenresscreenshotsmoviesrecommendationsachievementsrelease_datesupport_infobackgroundcontent_descriptors
0gameCounter-Strike100FalseNaNNaNPlay the world's number 1 online action game. ...Play the world's number 1 online action game. ...Play the world's number 1 online action game. ...NaNEnglish<strong>*</strong>, French<strong>*</st...https://steamcdn-a.akamaihd.net/steam/apps/10/...NaN{'minimum': '\\r\\n\\t\\t\\t<p><strong>Minimum:</st...{'minimum': 'Minimum: OS X Snow Leopard 10.6....{'minimum': 'Minimum: Linux Ubuntu 12.04, Dual...NaNNaNNaN['Valve']['Valve']NaN{'currency': 'GBP', 'initial': 719, 'final': 7...[7][{'name': 'default', 'title': 'Buy Counter-Str...{'windows': True, 'mac': True, 'linux': True}{'score': 88, 'url': 'https://www.metacritic.c...NaN[{'id': 1, 'description': 'Multi-player'}, {'i...[{'id': '1', 'description': 'Action'}][{'id': 0, 'path_thumbnail': 'https://steamcdn...NaN{'total': 66232}{'total': 0}{'coming_soon': False, 'date': '1 Nov, 2000'}{'url': 'http://steamcommunity.com/app/10', 'e...https://steamcdn-a.akamaihd.net/steam/apps/10/...{'ids': [2, 5], 'notes': 'Includes intense vio...
1gameTeam Fortress Classic200FalseNaNNaNOne of the most popular online action games of...One of the most popular online action games of...One of the most popular online action games of...NaNEnglish, French, German, Italian, Spanish - Sp...https://steamcdn-a.akamaihd.net/steam/apps/20/...NaN{'minimum': '\\r\\n\\t\\t\\t<p><strong>Minimum:</st...{'minimum': 'Minimum: OS X Snow Leopard 10.6....{'minimum': 'Minimum: Linux Ubuntu 12.04, Dual...NaNNaNNaN['Valve']['Valve']NaN{'currency': 'GBP', 'initial': 399, 'final': 3...[29][{'name': 'default', 'title': 'Buy Team Fortre...{'windows': True, 'mac': True, 'linux': True}NaNNaN[{'id': 1, 'description': 'Multi-player'}, {'i...[{'id': '1', 'description': 'Action'}][{'id': 0, 'path_thumbnail': 'https://steamcdn...NaN{'total': 2816}{'total': 0}{'coming_soon': False, 'date': '1 Apr, 1999'}{'url': '', 'email': ''}https://steamcdn-a.akamaihd.net/steam/apps/20/...{'ids': [2, 5], 'notes': 'Includes intense vio...
2gameDay of Defeat300FalseNaNNaNEnlist in an intense brand of Axis vs. Allied ...Enlist in an intense brand of Axis vs. Allied ...Enlist in an intense brand of Axis vs. Allied ...NaNEnglish, French, German, Italian, Spanish - Spainhttps://steamcdn-a.akamaihd.net/steam/apps/30/...http://www.dayofdefeat.com/{'minimum': '\\r\\n\\t\\t\\t<p><strong>Minimum:</st...{'minimum': 'Minimum: OS X Snow Leopard 10.6....{'minimum': 'Minimum: Linux Ubuntu 12.04, Dual...NaNNaNNaN['Valve']['Valve']NaN{'currency': 'GBP', 'initial': 399, 'final': 3...[30][{'name': 'default', 'title': 'Buy Day of Defe...{'windows': True, 'mac': True, 'linux': True}{'score': 79, 'url': 'https://www.metacritic.c...NaN[{'id': 1, 'description': 'Multi-player'}, {'i...[{'id': '1', 'description': 'Action'}][{'id': 0, 'path_thumbnail': 'https://steamcdn...NaN{'total': 2013}{'total': 0}{'coming_soon': False, 'date': '1 May, 2003'}{'url': '', 'email': ''}https://steamcdn-a.akamaihd.net/steam/apps/30/...{'ids': [], 'notes': None}
3gameDeathmatch Classic400FalseNaNNaNEnjoy fast-paced multiplayer gaming with Death...Enjoy fast-paced multiplayer gaming with Death...Enjoy fast-paced multiplayer gaming with Death...NaNEnglish, French, German, Italian, Spanish - Sp...https://steamcdn-a.akamaihd.net/steam/apps/40/...NaN{'minimum': '\\r\\n\\t\\t\\t<p><strong>Minimum:</st...{'minimum': 'Minimum: OS X Snow Leopard 10.6....{'minimum': 'Minimum: Linux Ubuntu 12.04, Dual...NaNNaNNaN['Valve']['Valve']NaN{'currency': 'GBP', 'initial': 399, 'final': 3...[31][{'name': 'default', 'title': 'Buy Deathmatch ...{'windows': True, 'mac': True, 'linux': True}NaNNaN[{'id': 1, 'description': 'Multi-player'}, {'i...[{'id': '1', 'description': 'Action'}][{'id': 0, 'path_thumbnail': 'https://steamcdn...NaN{'total': 942}{'total': 0}{'coming_soon': False, 'date': '1 Jun, 2001'}{'url': '', 'email': ''}https://steamcdn-a.akamaihd.net/steam/apps/40/...{'ids': [], 'notes': None}
4gameHalf-Life: Opposing Force500FalseNaNNaNReturn to the Black Mesa Research Facility as ...Return to the Black Mesa Research Facility as ...Return to the Black Mesa Research Facility as ...NaNEnglish, French, German, Koreanhttps://steamcdn-a.akamaihd.net/steam/apps/50/...NaN{'minimum': '\\r\\n\\t\\t\\t<p><strong>Minimum:</st...{'minimum': 'Minimum: OS X Snow Leopard 10.6....{'minimum': 'Minimum: Linux Ubuntu 12.04, Dual...NaNNaNNaN['Gearbox Software']['Valve']NaN{'currency': 'GBP', 'initial': 399, 'final': 3...[32][{'name': 'default', 'title': 'Buy Half-Life: ...{'windows': True, 'mac': True, 'linux': True}NaNNaN[{'id': 2, 'description': 'Single-player'}, {'...[{'id': '1', 'description': 'Action'}][{'id': 0, 'path_thumbnail': 'https://steamcdn...NaN{'total': 4402}{'total': 0}{'coming_soon': False, 'date': '1 Nov, 1999'}{'url': 'https://help.steampowered.com', 'emai...https://steamcdn-a.akamaihd.net/steam/apps/50/...{'ids': [], 'notes': None}
\n", "
" ], "text/plain": [ " type name steam_appid required_age is_free \\\n", "0 game Counter-Strike 10 0 False \n", "1 game Team Fortress Classic 20 0 False \n", "2 game Day of Defeat 30 0 False \n", "3 game Deathmatch Classic 40 0 False \n", "4 game Half-Life: Opposing Force 50 0 False \n", "\n", " controller_support dlc detailed_description \\\n", "0 NaN NaN Play the world's number 1 online action game. ... \n", "1 NaN NaN One of the most popular online action games of... \n", "2 NaN NaN Enlist in an intense brand of Axis vs. Allied ... \n", "3 NaN NaN Enjoy fast-paced multiplayer gaming with Death... \n", "4 NaN NaN Return to the Black Mesa Research Facility as ... \n", "\n", " about_the_game \\\n", "0 Play the world's number 1 online action game. ... \n", "1 One of the most popular online action games of... \n", "2 Enlist in an intense brand of Axis vs. Allied ... \n", "3 Enjoy fast-paced multiplayer gaming with Death... \n", "4 Return to the Black Mesa Research Facility as ... \n", "\n", " short_description fullgame \\\n", "0 Play the world's number 1 online action game. ... NaN \n", "1 One of the most popular online action games of... NaN \n", "2 Enlist in an intense brand of Axis vs. Allied ... NaN \n", "3 Enjoy fast-paced multiplayer gaming with Death... NaN \n", "4 Return to the Black Mesa Research Facility as ... NaN \n", "\n", " supported_languages \\\n", "0 English*, French*Minimum:Minimum:Minimum:Minimum:Minimum:\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
appidnamedeveloperpublisherscore_rankpositivenegativeuserscoreownersaverage_foreveraverage_2weeksmedian_forevermedian_2weekspriceinitialpricediscountlanguagesgenreccutags
010Counter-StrikeValveValveNaN1252193366020,000,000 .. 50,000,00011760143519999990English, French, German, Italian, Spanish - Sp...Action0{'Action': 5247, 'FPS': 4638, 'Multiplayer': 3...
120Team Fortress ClassicValveValveNaN333763402,000,000 .. 5,000,0001902504994990English, French, German, Italian, Spanish - Sp...Action0{'Action': 721, 'FPS': 289, 'Multiplayer': 241...
230Day of DefeatValveValveNaN345140505,000,000 .. 10,000,000150904994990English, French, German, Italian, Spanish - SpainAction0{'FPS': 769, 'World War II': 237, 'Multiplayer...
340Deathmatch ClassicValveValveNaN128827005,000,000 .. 10,000,000901204994990English, French, German, Italian, Spanish - Sp...Action0{'Action': 620, 'FPS': 132, 'Classic': 99, 'Mu...
450Half-Life: Opposing ForceGearbox SoftwareValveNaN529629505,000,000 .. 10,000,000381038904994990English, French, German, KoreanAction0{'FPS': 853, 'Action': 278, 'Classic': 227, 'S...
\n", "" ], "text/plain": [ " appid name developer publisher score_rank \\\n", "0 10 Counter-Strike Valve Valve NaN \n", "1 20 Team Fortress Classic Valve Valve NaN \n", "2 30 Day of Defeat Valve Valve NaN \n", "3 40 Deathmatch Classic Valve Valve NaN \n", "4 50 Half-Life: Opposing Force Gearbox Software Valve NaN \n", "\n", " positive negative userscore owners average_forever \\\n", "0 125219 3366 0 20,000,000 .. 50,000,000 11760 \n", "1 3337 634 0 2,000,000 .. 5,000,000 19 \n", "2 3451 405 0 5,000,000 .. 10,000,000 15 \n", "3 1288 270 0 5,000,000 .. 10,000,000 9 \n", "4 5296 295 0 5,000,000 .. 10,000,000 381 \n", "\n", " average_2weeks median_forever median_2weeks price initialprice \\\n", "0 1 435 1 999 999 \n", "1 0 25 0 499 499 \n", "2 0 9 0 499 499 \n", "3 0 12 0 499 499 \n", "4 0 389 0 499 499 \n", "\n", " discount languages genre ccu \\\n", "0 0 English, French, German, Italian, Spanish - Sp... Action 0 \n", "1 0 English, French, German, Italian, Spanish - Sp... Action 0 \n", "2 0 English, French, German, Italian, Spanish - Spain Action 0 \n", "3 0 English, French, German, Italian, Spanish - Sp... Action 0 \n", "4 0 English, French, German, Korean Action 0 \n", "\n", " tags \n", "0 {'Action': 5247, 'FPS': 4638, 'Multiplayer': 3... \n", "1 {'Action': 721, 'FPS': 289, 'Multiplayer': 241... \n", "2 {'FPS': 769, 'World War II': 237, 'Multiplayer... \n", "3 {'Action': 620, 'FPS': 132, 'Classic': 99, 'Mu... \n", "4 {'FPS': 853, 'Action': 278, 'Classic': 227, 'S... " ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# inspect downloaded steamspy data\n", "pd.read_csv('../data/download/steamspy_data.csv').head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Next Steps\n", "\n", "Here we have defined and demonstrated the download process used to generate the data sets. This was completed separately but the full, raw data can be found [on Kaggle](https://kaggle.com/nikdavis/steam-store-raw).\n", "\n", "We now have two tables of data with a variety of information about apps on the Steam store. From the Steam data it looks like there are some useful columns like `required_age`, `developers` and `genres` which we can eventually turn into features for analysis, and a `price_overview` column which may inform the success and sales of each game. The `owners` column of the SteamSpy data could be useful, however the [margin of error](https://steamspy.com/about) means data may not be accurate enough for meaningful analysis, we'll have to see what we can manage after cleaning. Instead we may have to use the `positive` and `negative` ratings or average play-time to create our metrics. There is also a `tags` column which appears to crossover with the `categories` and `genres` columns in the Steam data. We may wish to merge these, or keep one over the other.\n", "\n", "These are all decisions we'll come to in later stages of the project. With the data downloaded, this stage is now complete. In the next step, we'll take care of preparing and cleaning the data, readying a complete data set to use for analysis." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.3" } }, "nbformat": 4, "nbformat_minor": 2 }