Real-word Squid HTTP proxy hierarchy setup - Expat guide

Half a year back I posted about moving to Sweden. I built network and router setup to gain access to The Net. One of the obvious use cases for a connection is to playback various streaming media services. I have touced the subject of configuring Squid for YLE Areena earlier.

As an ex-patriate, I like to see and hear some of the media services I've used to consuming in Finland. Those stupid distribution agreements  make that unnecessarily difficult.

Spec

With my current setup, I need to access services:

• directly: Using no fancy routing, direct out of my Swedish IP-address. It's faster and less error-prone. This is majority of my HTTP-traffic.
• from Finland: To access geo-IP blocked services from a Finnish IP. I have my own server there and I'm running Squid on it.
• from my VPN-provider: To access selected geo-IP blocked services from a country of my choosing

Of course I can go to my browser settings and manually change settings to achieve any of those three, but what if I want to access all of them simultaneously? To get that, I started browsing Squid configuration directive reference. Unfortunately that is what it says, a reference manual. Some understanding about the subject would be needed for a reference manual to be useful.

Getting the facts

Luckily I have access to Safari Books Online. From that I found an on-line book called Squid: The Definitive Guide:

That has an useful chapter in it about configuring a Squid to talk to other Squids forming a hierarchy:

© for above excerpt: O'Reilly Media

Now I familiarized myself with Squid configuration directives like: cache_peer, cache_peer_access and always_direct.
... but. Having the book, reading it, reading the man-pages and lot of googling around left me puzzled. In the wild-wild-net there is lot of stumble/fall-situations and most people simply won't get the thing working and those who did, didn't configure their setup like I wanted to use it.

Darn! Yet again: if you want to get something done, you must figure it out by yourself. (again!)

Desired state

This is a digaram what I wanted to have:

Implementation

But how to get there? I kept stumbing with always_direct allow never and never_direct allow always -permutations while trying to declare some kind of ACLs or groupings which URLs should be processed by which proxy.

After tons and tons of failure & retry & fail again -loops, I got it right! This is my home (Sweden) proxy config targeting the two other Squids to break out of geo-IP -jail:

# 1) Define cache hierarchy
# Declare peers to whom redirect traffic to.
cache_peer proxy-us.vpnsecure.me parent 8080 0 no-query no-digest name=vpnsecure login=-me-here!-:-my-password-here!-
cache_peer my.own.server parent 3128 0 no-query no-digest name=mybox default

# 2) Define media URLs
# These URLs will be redirected to one of the peers.
acl media_usanetwork dstdomain www.usanetwork.com www.ipaddress.com
acl media_netflix dstdomain .netflix.com
acl media_yle dstdomain .yle.fi areenapmdfin-a.akamaihd.net yletv-lh.akamaihd.net .kaltura.com www.ip-adress.com
acl media_ruutu dstdomain ruutu.fi geoblock.nmxbapi.fi
acl media_katsomo dstdomain .katsomo.fi
acl media_cmore dstdomain cmore.fi

# 3) Pair media URLs and their destination peers.
# Everything not mentioned here goes directly out of this Squid.
cache_peer_access vpnsecure allow media_usanetwork
cache_peer_access vpnsecure allow media_netflix
cache_peer_access mybox allow media_yle
cache_peer_access mybox allow media_ruutu
cache_peer_access mybox allow media_katsomo
cache_peer_access mybox allow media_cmore
never_direct allow media_usanetwork
never_direct allow media_yle
never_direct allow media_ruutu
never_direct allow media_katsomo
never_direct allow media_cmore
never_direct allow media_netflix

# 4) end. Done!
request_header_access X-Forwarded-For deny all

Simple, isn't it!
Nope. Not really.

Notes / Explanation

Noteworthy issues from above config:

• On my web browser, I configure it to use the home proxy for all traffic. That simply makes my life easy. All traffic goes trough there and my local Squid makes the decisions what and where to go next, or not.
• In above configuration:
  • I declare two peers (cache_peer), how and where to access them.
  • I declare ACLs by URL's destination domain (acl dstdomain). It's fast and easy!
  • I associate ACLs with peers (cache_peer_access) to get HTTP-routing
  • I make sure, that any traffic destined to be routed, does not exit directly from Sweden (never_direct allow). Slipping with that will be disastrous for geo-IP checks!
  • I don't declare my traffic to be originating from a HTTP-proxy (request_header_access X-Forwarded-For deny). Some media services follow up to the real client-address and do their geo-IP checks on that.
• The VPN-service, I'm using is VPNSecure or VPN.S as they like to be called nowadays. Two reasons:
  1. No mandatory softare installation needed. You can just configure your browser to use their proxy with proxy authentication. And if you are using Google Chrome as your browser, there is an VPN.S extension for super easy configuration!
  2. Price. It's very competitive. You don't have to pay the list prices. Look around and you can get ridiculous discounts on their announces list prices.
• My Finnish proxy is just out-of-the-box Squid with ACLs allowing my primary proxy to access it via Internet. Other than default settings, has request_header_access X-Forwarded-For deny in it.
• For testing my setup, I'm using following display-your-IP-address -services:
  • https://www.ipaddress.com/
  • https://www.ip-adress.com/
  • http://whatismyipaddress.com/
• Finnish services C More and MTV Katsomo are pretty much the same thing, share user accounts and in future are merging more and more. To have C More working, both ACLs need to declared.
• YLE Areena is using Akamai CDN extensively and they tend to change.
• Ruutu is using external service for geo-IP for added accuracy. They compare the results and assume the worst! Redirecting both is necessary to keep their checks happy.
• Netflix is using their own state-of-the-art CDN, which is very easy to configure. Regardless what Mr. Hastings publicly, they only care that the monthly recurring credit card payment passes. Using different countries for Netflix produces surprising results for availability of content.

Finally

This is working nicely!

Now moving from one media service to another is seamless and they can run simultaneously. Not that I typically would be watching two or more at the same time, but ... if I wanted to, I could!