Router-based ad-blocking. Possible? Totally!

[Edvard]

Hey... psssst!  
Ever wanted to block internet advertising from within your router? "Why would I want to do such a thing?", you may ask.
Well, whether you want ad-blocking for your mobile devices while at home (most mobile browsers don't allow plug-ins, including AdBlock), you want to help your WiFi guests escape ads while they're logged in at your place, or you just want a fun evening project hacking around in your router, just follow these steps to ad-blocking bliss...


I got this idea from a few websites and forum posts out there that have elaborate scripts for parsing the above hosts file links, cobbling them into a custom file, and doing it on a regular schedule to keep the hosts file up to date.  I wanted to keep it simple, so I just created a hosts file manually and uploaded it to my router, which worked just fine.

First, a few caveats:  
1 - It only works on DD-WRT and it's step-children, and
2 - It's really nothing more than feeding your router an extra 'hosts' file (for more information on that, see HERE, and HERE and um... HERE about that...), so nothing fancy.  
3- You have to have "Local DNS" and "DNSMasq" enabled on your router so it can act as a proxy for your network's DNS requests (we'll go over that in this post).

More caveats:  I still don't know how to block Youtube ads, and apparently Internet Explorer doesn't really dig URL redirects to empty ether, so some some good folks have come up with something called Pixelserv that is simply a webserver whose sole purpose in life is to serve back exactly one pixel, so IE doesn't get stuck in a loop actually expecting something back when it calls to the abyss for a non-existent ad URL.  

Links to articles describing installing and using Pixelserv and advanced host-file voodoo hijinx at the bottom.

So, let's get started...

First of all, log in to your router and select the "Administration" tab.

(I am using a Buffalo WZR-HP-G300NH with a branded version of DD-WRT.  Your interface may differ somewhat.)

It will ask you to log in:


From there, scroll down until you get to the part labeled "JFFS2"

Enable it, and if this is the first time you are doing this, enable "Clean JFFS2" also.

Apply settings:

Don't reboot yet!  We will do that later.

Go to the "Services" tab:


and scroll down to "DNSMasq":


Type in as shown:

[Select]

addn-hosts=/jffs/hosts

NOW comes the hacker part:

A 'hosts' file is a list of addresses and URLs arranged in a [address] [space] [URL] format, like so:

[Select]

127.0.0.1 sleazyadserver.com
127.0.0.1 naughtyadserver.net
127.0.0.1 badguys247.org

So, with your favorite text editor, copy and paste in the host-file entries from the links above, or from the links HERE, plus any that you would like to include, and edit the redirect address to your liking.  Most folks will use 127.0.0.1, some use 0.0.0.0, use whatever tickles your fancy, but remember to save the file with Unix line endings and plain ol' ascii text.


Call it 'hosts.txt' for now.

Now, using 'scp' (if on Linux or BSD) or 'pscp' (PuTTY's implementation of scp on Windows), upload the file to your router.
The syntax for scp is: [scp or path to pscp] [path to the hosts file you created] root@[your router's address]:/path/to/somewhere

From Linux, if I put the hosts file in my home directory, and my router is at 192.168.11.1, it looks like this:

[Select]

scp ~/hosts.txt [email protected]:/jffs/hosts

On Windows, if I put the file on my Desktop, it might look like this in a command window (Start Menu -> Run.. -> 'cmd'):

[Select]

"C:\Program Files\PuTTY\pscp.exe" "C:\users\edvard\Desktop\hosts.txt" [email protected]:/jffs/hosts

Next, log back into your router via the web interface, click the "Administration" tab, go to the bottom of the page and reboot!


Now, open a command window and try to ping one of the adservers listed in the hosts file:

[Select]

ping www.clickhouse.com

If you get a response like this:

[Select]

PING www.clickhouse.com (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.019 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.032 ms
64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.038 ms

--- www.clickhouse.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.019/0.029/0.038/0.010 ms

... then you're successful.  If not, something's not right, and I suggest you go through the steps again, or maybe I got something wrong, and I need your help to de-bug the process.  Happy ad-free routering!  


References and resources:

Ad blocking from the DD-WRT Wiki, complete with complicated automatic update script and everything:
http://www.dd-wrt.co...ndex.php/Ad_blocking

How-to Geek article on ad-blocking with the Pixelserv:
http://www.howtogeek...pixelserv-on-dd-wrt/

Turn your Raspberry Pi into a wireless access point with ad-blocking:
https://learn.adafru...ccess-point/overview
https://learn.adafru...ccess-point/overview

[lanux128]

thanks for the write-up, Edvard!

[Deozaan]

Awesome! Thanks for posting this!

[4wd]

I mentioned the one I use back here.

Has been working without a problem for the last 15 months, auto-updates itself.

[Edvard]

Wow! I didn't catch you posting that.  I also read the post you linked to (http://www.linksysin...an-adblocking.68464/), but what I wanted was to start small, you know, basic bare-bones adblocking with a hosts file in your router.  How simple is that?  And the payoff is immediate.  Once folks catch on to the possibilities, THEN we can look into auto-updating scripts, running Pixelsrev, parsing multiple host files auto-magically, etc.

Thanks for reminding me of that post though, it was part of the inspiration and I couldn't find it when I went back looking for references for my post.  

[Deozaan]

I get stuck at this step. I can't seem to connect to my router using PuTTy or WinSCP or pscp.

Now, using 'scp' (if on Linux or BSD) or 'pscp' (PuTTY's implementation of scp on Windows), upload the file to your router.
The syntax for scp is: [scp or path to pscp] [path to the hosts file you created] root@[your router's address]:/path/to/somewhere

From Linux, if I put the hosts file in my home directory, and my router is at 192.168.11.1, it looks like this:

[Select]

scp ~/hosts.txt [email protected]:/jffs/hosts

On Windows, if I put the file on my Desktop, it might look like this in a command window (Start Menu -> Run.. -> 'cmd'):

[Select]

"C:\Program Files\PuTTY\pscp.exe" "C:\users\edvard\Desktop\hosts.txt" [email protected]:/jffs/hosts

-Edvard (October 30, 2014, 12:52 AM)

[4wd]

THEN we can look into auto-updating scripts, running Pixelsrev, parsing multiple host files auto-magically, etc.

-Edvard (October 30, 2014, 08:30 PM)

I'll look at doing a reinstall with pictures for you on my router, Tomato USB but it should work for the other DD-WRT varieties.  Only thing different would be where I install it as I have a permanently plugged in flash drive on the router.

[4wd]

Continuing on with Edvard's thread since it's related, (hope you don't mind Edvard), we'll use a different method of getting the same effect.

I've done this using my router which is running Tomato by shibby but it should be the same also, (with interface differences), for the DD-WRT family, basically as per Edvard said in his OP.

We're going to install Almazick' AdBlock which also has a Web based GUI, you can get it at GitHub if you want to do it manually yourself.

NOTE: I have downloaded the zip file from the above site and are hosting it on one of my VPSs for two reasons:

• The wget command on the router can not handle HTTPS URLs, so it can't get the file from GitHub.
• I have rezipped the file to remove the folder structure, it's not needed and it makes the process simpler.

If you don't trust me feel free to download the original from GitHub and either use it or compare it with the one I'm hosting - don't worry, I'm incapable of being offended  

NOTE: By default, the IP for the pixelserver that gets run is x.x.x.254, ie. if your router is 192.168.1.1, then the pixelserver will be 192.168.1.254   If this is a problem for anyone, let me know and I'll edit the config file in the archive to change it to something more obscure or you can change it after it's installed, (see further down at the web interface).


I've tried to make this as painless as possible by not having to use any form of SSH/ftp or anything other than the web interface of your router.

Before we start, refer back to Edvard's post and enable JFFS on your router, there's no need to reboot.

1) Log in to your routers web interface and go to, (under Tomato), Tools->System Commands.
2) Paste the following lines into the Command text field.

Code: Text [Select]

1. cd /jffs
2. mkdir adblock
3. cd adblock
4. wget http://167.160.167.146/adblock.zip
5. unzip adblock.zip
6. chmod +x *.sh ; chmod +x pixelserv
7. ./adblock.sh

You should have something like this on the screen:



3) Press the Execute button.

It will take anywhere from a few seconds to a minute or so, depends on your connection speed and how fast your routers CPU is.

At the end of that time the output from the commands should appear, like so:



Congratulations!  It's installed  and running ... pretty painless.


What happened:

• Change to the /jffs directory
• Create the adblock directory and change to it.
• Fetch the adblock.zip archive
• Unzip it.
• Set the executable attribute on the *.sh and pixelserver files.
• Execute the adblock.sh script which:
  • Reads its config file.
  • Creates a symbolic link so you can access the web interface, (more later).
  • Downloads the hosts file from the mvps.org site (see here).
  • Generates the blocklist to be used by dnsmasq.
  • Starts the pixelserver (this is a minimal webserver that just serves a single pixel).
  • Write dnsmasq.custom which is a config file for dnsmasq that tells it to call the blocklist (which is written as a dnsmasq config file - convoluted isn't it?).
  • Finally restarts dnsmasq so it can use the new config files.
• adblock.sh then exits as it's no longer needed, all the work is done by dnsmasq from this point.

The Web Interface
Now that adblock.sh has been installed and has run at least once, you should be able to access the web interface from where you can control it.

There is a caveat:

• It seems you can't only use HTTPS to access your routers interface.  You need to use HTTP or HTTPS & HTTP.

This screwed me up to start with as I always use only HTTPS, if you don't mind losing the web interface I'll show you how to control adblock.sh using your routers facilities later.

You can access the adblock web interface by going to: http://<router IP>:<port>/user/adblock.sh  (there's no need for a port number if your routers default interface port is 80)

You should end up with this:



If it says logging: DISABLED just click on DISABLED and it will start displaying output in the section below, refreshing every 120 seconds.

The adblock actions are as follows:

• force           - forces an update download of the hosts file, normally it won't do an update if it's too soon after the last one.
• start/update - starts adblock.sh or causes an update download if it's running.  You will be told if the update didn't happen because it was too soon after the last.
• restart         - self-explanatory
• stop            - self-explanatory
• edit lists       - allows you to edit/save the BLACKLIST and WHITELIST, these are not the same as the BLOCKLIST.  The BLOCKLIST is generated by adblock.sh from downloaded hosts lists and contains dnsmasq commands.  The BLACKLIST/WHITELIST are just a list of domains that you always want to block or allow.
• edit config    - edit the adblock.sh config file.  Unless you know what you're doing there's only one value in there you would need to change, and that's the value that concerns the IP of the pixelserver.  By default the last octet of the IP the pixelserver uses is 254, eg. if your router is 192.168.1.32 then the pixelserver IP will be 192.168.1.254 - if this happens to clash with another computer/etc, then you can change it here, save, and restart the adblock.sh script.

The big test:

Open a CLI and type tracert ads.fairfax.com.au, as in the image below, you should see the domain be resolved to the IP of pixelserver and after a refresh the web interface will show that the host has been blocked.

If you don't see any logging info, go to Advanced->DHCP/DNS and enter log-queries into the Dnsmasq Custom Configuration box, then hit the Save button.




Surviving a router reboot:

At this point in time, adblock.sh won't survive router reboots, ie. when the router starts next time, adblock.sh will need to be run again - until that time, there is no adblocking nor is there a web interface from which to start it.

So, what do we do?

We use Scripts:

In your router interface, go to Administration->Scripts and click the WAN Up tab, enter /jffs/adblock/adblock.sh in the text area and hit the Save button down the bottom:



Now when your router establishes a connection to the WAN, adblock.sh will run and set itself up again.


Automatic updates:

Rather than have to manually update the BLOCKLIST via the adblock web interface, let's do it automatically.

Head to Administration->Scheduler, enable one of the Custom schedules and fill it out as below, set a convenient time for the update to occur (once a week is more than enough):



Remember to hit the Save button after making any changes.


Toggling adblock On/Off:

You want to allow ad servers for some reason, (you need your head read), you could fire up the web interface or you could use one more nifty feature of your router.

Administration->Buttons/LED allows you to set the function of the WPS button depending on how long it's held down.  You might already be using it for something so treat this as an example:



Remember to hit the Save button after making any changes.

If you used the same settings as I did in the image, when you press the WPS button for less than 2 seconds it will toggle the state of adblock.sh - if it's On it goes Off and vice versa.  You'll see its status change to down or up in the web interface depending on it's condition.

Now any connection to an ad server will go through, press the button again and they're blocked.  Just a small note, it sometimes takes a couple of button presses to get adblock.sh to toggle, maybe my button is dodgy ... who knows ... I don't turn it off anyway.

That about wraps it up.

A couple of things that may be of interest:

• If you want to use an updated version of the pixelserver, change adblock.zip to adblock_HZ11.zip in the wget and unzip lines in step 2 - it came from here if you want to check it out.
• In the adblock_HZ11.zip archive there is a file called README_pix.md which explains what all the three-letter-acronyms in the pixelserver status display are.
• The config file, (/jffs/adblock/config), is well commented and easily understood.
• If you want to enable some of the other hosts lists in the configuration I suggest you download the files first and see how big they are, one of them is quite large and you may not have the capacity in your router to handle it.

For interest, these are the hosts lists that are already in adblock.sh config, only the first one is enabled by default:

• MVPS HOSTS (~600k) [default]: http://winhelp2002.mvps.org/hosts.txt
• pgl.yoyo.org (~64k): http://pgl.yoyo.org/...p;mimetype=plaintext
• Hosts File Project (~2.7M!): http://hostsfile.mine.nu/Hosts
• The Cameleon Project (~619k): http://sysctl.org/cameleon/hosts
• AdAway mobile ads (~15k): http://adaway.org/hosts.txt
• hpHosts ad/tracking servers (~790k): http://hosts-file.net/ad_servers.asp
• hpHosts ad/tracking/malicious servers (~30M! replaces hpHosts ad/tracking list): http://hosts-file.net/download/hosts.txt, http://hosts-file.ne.../hphosts-partial.asp
• MalwareDomainList.com (~42k): http://www.malwaredo.../hostslist/hosts.txt

I recommend that you do not enable the second last one because at approx. 30MB I doubt whether you'd have enough space in your router to handle it. 

[Edvard]

Continuing on with Edvard's thread since it's related, (hope you don't mind Edvard)
...

-4wd (November 30, 2014, 04:31 AM)

I don't mind at all!  
Awesome post and I'm thinking I may just do what you've done, as it looks pretty painless.  Many others I looked at were a bit hacked-together and I was looking for something clean.  You've put together what looks like a clean package and relatively painless process, thanks!
 

[4wd]

Thanks!

By default they usually get you to install it in /tmp which means that everything disappears after a router boot.  They get around that by downloading it all again when the WAN comes up.

The script will work from anywhere, so you can install it to a flash drive if your router has the capability - just change any paths in my post above to the relevant one and you should be OK.

If your router doesn't have JFFS you can install it to /tmp as per originally and use the commands in step 2 as the WAN Up script - change paths to suit.

I tested my method above about 3 or 4 times as I was trying to put it in a suitable location, it seemed to work every time but let me know if you hit a problem and I may be able to help.

[4wd]

If you don't have a firmware in your router that has JFFS capability, (eg. half of the shibby firmwares don't have the capability), then you can install adblock into the /tmp area of your routers flash RAM.

shibby Tomato builds

There's a downside to this, anything in /tmp is deleted when the router restarts ... but we can get around it  

Refer back to the images in my post as they're the same, the only difference is the paths used which I'll cover here:

Step 1) Log in to your routers web interface and go to, (under Tomato), Tools->System Commands.
Step 2) Paste the following lines into the Command text field.

Code: Text [Select]

1. cd /tmp
2. wget http://167.160.167.146/adblock.zip
3. unzip adblock.zip
4. chmod +x *.sh ; chmod +x pixelserv
5. ./adblock.sh

Step 3) Click the Execute button.

After it's executed you should have a functioning router based adblocker which you can check by using the same test as in my post.

To get it to survive restarts we need to change the WAN Up script to the following:



As you can see, all that will happen is when the router makes a connection to the WAN is that it will download and install the adblock script again.

Everything else in my post, (Automatic Updates, Adblock Toggle), you can also do just remember to change the paths from /jffs/adblock/adblock.sh to /tmp/adblock.sh


NOTE: At this point I should mention that you should download the adblock.zip file and host it somewhere so you're not relying on my VPS all the time.  Anywhere that can be accessed by the wget command in the router is fine, this means accessible by either HTTP or FTP protocols - not HTTPS, SFTP, or FTPS.
It will also allow you update the archive yourself with either a later adblock GitHub version or pixelserver version.


To update the adblock scripts/pixelserver:

If you're using /tmp
1) Update your adblock.zip archive.
2) Reboot your router.


If you're using /jffs
1) Update your adblock.zip archive.
2) Stop adblock, (via web interface or toggle).
3) Use these commands in the Tools->System Commands

Code: Text [Select]

1. cd /jffs/adblock
2. rm -f adblock.zip
3. wget http://167.160.167.146/adblock.zip
4. unzip -o adblock.zip
5. chmod +x *.sh ; chmod +x pixelserv
6. ./adblock.sh

Naturally, change the IP above to reflect wherever you have your copy of the adblock.zip file.