steam content delivery system - CoursesEssaydatesCom

steam content delivery system

Subscriptions

Close


Navigate

  • Videos
  • Features
  • Reviews
  • Ars Approved
  • RSS Feeds
  • Mobile Site
  • About Ars
  • Staff Directory
  • Contact Us
  • Advertise with Ars
  • Reprints


Filter by topic

  • Biz & IT
  • Tech
  • Science
  • Policy
  • Cars
  • Gaming & Culture
  • Forums


Settings

Front page layout


Grid


List


Wide

Site theme

Black on white

White on black

Sign in


Comment activity

Sign up or login to join the discussions!

Sign up to comment and more
Sign up

GIMME THE CACHE —

Building a local Steam caching server to ease the bandwidth blues

A bit of Linux, a bit of Nginx, and boom—fast and free Steam downloads.

Lee Hutchinson

Enlarge
Aurich Lawson

reader comments
136

Share this story

If you play games on PC, where God intended them to be played, chances are you’ve got Steam installed. Since its troubled launch in 2003, Valve’s publishing platform has gone from a thing we had to grudgingly put up with in order to play Half Life 2 to the most popular digital game distribution tool on the planet. Most of us—me included—interact with it pretty much daily.

But as game distribution was shifting from physical to digital, ISPs also began implementing data caps—usually under the guise of “network management” (though anyone who thinks caps aren’t a pure revenue play should send me an e-mail, because I’ve got a bridge to sell you, cheap!). Steam is fast, Steam is easy, and Steam is ubiquitous—but if you’ve just rebuilt a PC or reinstalled an OS and you need to reinstall your games, Steam will obligingly help you put a giant dent in your cap—very quickly.

But there’s an alternative to having to re-download all your Steam games from the Internet: you can set up a local Steam caching server, so that once you download something, you’ve got it on your LAN instead of having to reach for it across the net and incur usage fees.

How this works

The idea is simple: when your computer’s Steam client needs to install or update an application, it contacts one or more of Valve’s SteamPipe content distribution servers. Valve has partnered with ISPs across the world so that most folks have SteamPipe content servers near them (“near” in terms of both physical distance and also network hops). This saves ISPs money on transit and peering .

SteamPipe is used to deliver what the client needs, be it a whole game or just an update, in roughly megabyte-size chunks. (Chunking like this allows developers to publish updates without having to push a whole new game package—they just invalidate old chunks and upload new ones.) As Valve points out on the SteamPipe developer community page , SteamPipe uses plain ol’ HTTP rather than a proprietary protocol. And that gives us the opportunity to stick our fingers into the process and mess with it.

How things will work with our Steam caching server if what we want isn’t in cache.
How things will work with our Steam caching server if what we want isn’t in cache.

Recall for a moment how a caching Web server works: a user hits a page, and the server checks its cache to see if the cache has what the user needs in it. If the cache does, the server delivers those objects directly. If not, the caching Web server forwards the request to a backend, retrieves whatever the user needs from that backend, and delivers it. Then the Web server stores the objects in cache for next time so it doesn’t have to bother the backend again.

What if we turn that process sort of inside out? What if we set up a caching Web server like Nginx locally? What if, instead of using the server as a reverse-proxy to cache responses for incoming requests from the Internet for a particular Web site… we use it as forward proxy (or just a regular proxy, I suppose) and cache the responses to our outgoing requests for Steam content?

It sounds like we’re inverting what a reverse-proxying Web server is supposed to be for and using it like a normal Squid-style proxy—but as it turns out, this actually works great.

How things will work with our Steam caching server if what we want <em>is</em> in cache.
How things will work with our Steam caching server if what we want is in cache.

Did I say “great”? I meant “almost great.” There’s another piece of the puzzle. We need to screw around a bit with DNS for this to actually work. Steam doesn’t have a “check my local server first” option to make this caching thing an automated process, so we’ll need to add DNS entries to fake Steam into thinking that its SteamPipe content server actually is the local cache server we’re going to build.

All roads lead to Rome, but this is our road

Fortunately, none of this is actually as complicated as it sounds—and, as we’ll see, there are pre-built Docker images to do all of this for you, if you don’t want to get your hands dirty.

But before we proceed any farther, we need to go over exactly what you need for your local Steam cache server. As with so many projects, you can take a huge variety of paths to get to the destination; you can do this with nothing more than Windows and your gaming PC, or you can use a Windows or Linux virtual machine, or you can use an actual physical Windows or Linux server. You can keep your Steam cache on local disk or on a network share. You can use whatever application you want, provided it can forward and cache HTTP requests.

For this guide, we’re going to look at two specific configurations: Linux (specifically Ubuntu 16.04 LTS Server) with Docker for the easy way; and Linux (again, Ubuntu 16.04 LTS Server) on bare metal for the DIY way. Both configurations will store the actual cache depot on a network share—I set my depot up using iSCSI for a number of reasons external to this guide, but there’s nothing wrong with NFS (or even Samba, I guess). If you have a few terabytes of disk space, you can also put your depot local on your server. It’s all up to you and what works for you, but this guide will talk about putting your depot on a NAS and accessing it via a mounted NFS share.

Logos, logos everywhere!
Logos, logos everywhere!

The star of our show in both configurations is going to be Nginx, a fast and powerful Web server that runs a significant chunk of the World Wide Web (it’s second in popularity only to Apache, the Microsoft Word of Web servers). Nginx’s ability to maintain its own local content cache from an upstream source is what makes this entire process work. If Nginx sounds familiar to you, it’s because we’ve written about using it at great length with our Web Served series (and, yes, those articles are in dire need of update—that’s on my 2017 to-do list!).

Other tools could be employed here, too—most notably, a standard proxy server like Squid. And, in fact, guides are out there that tell you how to configure Squid for just that . But we’re going with Nginx primarily because of all the documentation out there to do it this way, and also because there are Docker images of working configurations we can pick apart and use.

For the DNS portion, you can fall back on old-fashioned host file editing or more modern (and flexible) actual DNS configuration. If you have a local instance of dnsmasq or bind running on your LAN—say, on your NAT router or something—that’s where you’ll be working. If this sounds scary or complicated, fear not—the “easy way” method we’re about to describe will take care of everything for you. In the DIY section, we’ll be setting up bind and using that.

reader comments
136

Share this story

Lee Hutchinson
Lee is the Senior Technology Editor at Ars and oversees gadget, automotive, IT, and culture content. He also knows stuff about enterprise storage, security, and manned space flight. Lee is based in Houston, TX.
Email [email protected]

You must login or create an account to comment.

Channel Ars Technica

Previous story Next story

PC Games news and reviews from PCGamesN.com

Jeremy Peel
Jeremy Peel

What is SteamPipe?

Steam

This week, Team Fortress 2 became one of the last games to switch over from Steam’s old content delivery system to SteamPipe. Once the switch-over’s complete, the game will boot more promptly and maps will load a little faster – but you probably won’t register the difference any more than you do the face of the man who brings your Amazon parcels to the door.

Before then, though, you’ll have to sit through a 20 to 30 minute “conversion process”. But what is SteamPipe? What’s it for, and what does it mean for your games? 

There’s very little documentation available around SteamPipe: just a couple of wiki posts. But talking to developers and those close to Steam, we can get a clearer idea of what SteamPipe, is, and what it means for PC gamers going forward.

The first thing to understand is that SteamPipe uses HTTP to transfer game data, replacing the old Steam2 content delivery system. This is important for a few reasons: it means that Steam data can fly more easily between firewalls, and makes it easier for Valve to use third party HTTP caches and content delivery networks to speed up downloads.

At a more granular level, SteamPipe also allows for ‘delta updating’ – meaning that only changes to code need to be updated, which should make for quicker and easier patching. You’ll also notice that you’re able to browse the file structure of games downloaded via SteamPipe, rather than having them hidden away in .gcf files.

SteamPipe has also allowed for a new feature within Steam: the ability to set download rates . But it’s on the developer side of things that the new system will really make itself known.

In the traditional Steam update process, there’s a period of ‘approval’ – after developers submit their patch to Valve, and before it arrives on Steam. You might imagine that a Microsoft-style certification process is happening in the interim, in which Valve set about testing the updates for bugs. But Garry ‘Garry’s Mod’ Newman says that’s not the case at all:

“When I send the update to Valve they’re putting it on Steam,” he explained in a blog post last year . “The time delay is the delay between me uploading the update – sending the email – then someone at Valve getting into work, checking their email, and pushing the update out to the hundreds of Steam content servers.”

This manual phase of your average Steam update is the job of Valve’s technical account managers, who assign content servers to updates based on their geographical location, game size and popularity.

It’s a bottleneck and barrier: one in which Valve stand between game developers and their audience .

SteamPipe changes that. In practical terms, the new delivery system will mean that developers are able to upload and issue patches themselves, without any manual intervention whatsoever. They’ll be able to push out updates at opportune moments and in line with announcements. They’ll be able to warn players about the larger ones beforehand, and monitor and fix any problems at their own pace.

SteamPipe also gives flexibility within Steam for developers to branch their code and manage access to betas without hosting their own content servers. And code is slightly more secure: old versions of their games are now unavailable to curious eyes.

But SteamPipe comes with an inconvenience. If you’re a regular user of Steam, you’ll have seen the pop-up explaining that Steam has to update the game to a new content system. That is the conversion process; a new version of the game is created from the old files, before the previous version is deleted. The update process can take a long time: Valve themselves quote at least a 15 minute wait when you next start Team Fortress 2.

But it’ll be worth it. SteamPipe is the future of patching on PC. Now every time Dustbowl loads a little bit faster than you expected, you know what to thank.

Read More
TF 2 system requirements
Best FPS shooters on PC
Best multiplayer games on PC
Buy Now Team Fortress 2 tile

Promoted

How to master Monster Hunter: World’s deep combat system

How to master Monster Hunter: World’s deep combat system

7 reasons you should return to RuneScape and Old School RuneScape

7 reasons you should return to RuneScape and Old School RuneScape

Why a visit to Monster Hunter: World should be your videogame summer vacation

Why a visit to Monster Hunter: World should be your videogame summer vacation

About
Powered by Network-N