Brian Boll – Personal and technical musings

Block Ads and Tracking with Hosts File and NoScript

Adblock Plus

Browser plugin solutions for blocking ads such as Adblock Plus have become increasingly popular; at the time of this post ABP is the most used Firefox extension. For the uninitiated, ad blocking plugins work by offering various filters to block your browser from downloading from certain domains over the HTTP and HTTPS protocols. At first glance this seems like the ultimate solution and certainly is a good one but it has a few notable and often overlooked drawbacks compared to other solutions.

Hosts File and Advantages

The hosts file is a plain text file that resides on your computer and serves your operating system with mapping hostnames to IP addresses. That definition contains an inline explanation as to why editing your hosts file has more advantages than using ad blocking plugins. For one, because the hosts file serves your operating system at a native level, it has a performance edge compared to ad blocks happening at the browser level. Plugins can be a large burden on already resource intensive browsers due to their runtime nature as well as bugs that result in excess resource hogging. Hosts file blocking can be used with relative effectiveness against known malicious domains that may attempt to compromise your computer through other channels. Browser level blocking deals solely with the HTTP and HTTPS protocols whereas hosts file changes block all connections across all TCP/IP protocols including SSH and FTP. On top of all of that, this can be an additional method to disable or hinder tracking on top of solutions offered by software like ABP. It is important to note that there simple workarounds such as using a hardcoded IP address rather than a domain to launch an attack; therefore, as always, a motivated attacker using a targeted attack will not be deterred.

Getting Started

The first step is to find your hosts file depending on your operating system which you can edit using a text editor such as Notepad.

File Location and Editing

Note: You may lack permissions to save to the directory and will be therefore required to create a new file outside the directory and replace it. Be sure to copy the old hosts file just in case. Also be sure to save as a plain text file—a file with no extension, not even a “.txt” extension. I used Notepad++ to achieve this.

Windows location: C:\Windows\System32\drivers\etc\hosts
Linux location: \etc\hosts
OSX location: \private\etc\hosts

Simple Hosts File Edit

For this demonstration I am going to add just one rule to the hosts file. By adding the following line, the domain “ad.doubleclick.net” will resolve to 0.0.0.0 which is your local machine. This means that anytime that your browser is prompted to download anything from “ad.doubleclick.net” it will block the request and resolve it to your local machine, thereby not displaying the ad.

Simple addition of rule to block DoubleClick

I use 0.0.0.0 instead of 127.0.0.1 because I believe it performs a bit faster since 127.0.0.1 waits for a timeout and I often use 127.0.0.1 for development purposes.

Possible Anti-virus Conflicts

Upon saving the hosts file you may run into a problem with anti-virus software detecting a possible hijacking. A relatively common form of attack is to indeed edit the hosts file and resolve domains to malicious addresses; therefore, Windows Defender immediately comes to my aid. In order to make changes, you have to first add the file in whatever anti-viruses exclusion list. Below is the navigation in Windows Defender where I would click ‘Add’ and it would appear below the input in the list of exclusions.

Windows Defender Navigation

NoScript

NoScript is a browser plugin much like ABP in that it blocks content similarly. The difference is that by default NoScript blocks all JavaScript and other extra content. In order to use NoScript you have to individually whitelist content, causing your browser to reload the page and run accordingly. This is done using the dropdown and click-to-allow interface.

Example of the NoScript interface

There is a noticeable advantage to security and performance by allowing you to decide what content you see from the webpage the first time you visit. Performance is enhanced since you’ve limited the amount of scripts running to only those you allow and deem important. Security is present because you get to personally allow content rather than it just being downloaded automatically without your consent or even knowledge. This allows you time to look up what you’re allowing before giving consent to run.

So-Called Breaking Browsing Experience

Developers might be displeased to think that you are viewing their content how you deem fit but NoScript certainly gives the power back to the user who is ultimately vulnerable otherwise. The amount of content that must be whitelisted often reflects on the reputation of the site itself. For instance accessing Wikipedia might only require me to quickly click-to-whitelist “wikipedia.org” and “wikimedia.org”.

However accessing any sort of reputable or otherwise news publication may involve me deciding what to allow from a list of 10+ requests. Adblock Plus and NoScript “break” the browsing experience in the sense that the developer controls what content you allow. Adblock Plus simply breaks content it deems as ads while NoScript breaks most of the extra content on the site and allows you to rebuild it yourself by what you allow. You can however easily allow all scripts on the page with NoScript should you tire of dealing with the system.

ABP, NoScript, Hosts Combination Benchmarks

The reason I use a combination of NoScript and my hosts file is because it offers a considerable performance boost as well as the most safety. As browsers in general are huge resource burdens it is rather easy to resist the temptation to use both NoScript and Adblock Plus plugins. NoScript uses less resources than Adblock Plus, allows less scripts to run by default, and uses less bandwidth because of this. Below is a table showing the bandwidth benchmarks for a few possible configurations, MVPS is the name of a popular Hosts file that blocks known ad servers.

A few of the bandwidth saving percentages highlighted in red. Source: IANIX

Here we see the ideal configuration for bandwidth and security is using NoScript in conjunction with hosts file blocking to edge out the most bandwidth efficiency possible of the combinations.

Importance of Ads

It is good to remember that ads are an important source of revenue for sites and by allowing them to run you are already supporting them. I stress an emphasis on fair and reasonable ad allowance and for that reason I myself do not block tracking from Google’s DoubleClick or AdSense. However, I always believe it is important to exercise your freedom to maintain the level of privacy and security that allows us all to comfortably use the web.

It’s important to decide if editing your hosts file is for you as it may require upkeep and hinder browsing experience depending on whether you plan on using it for simply blocking ads or as a shortcut to DNS lookup for some of your most visited sites. If you plan on using a large hosts file of your own or a community one such as MVPS HOSTS which can be found here, it is necessary to disable the DNS Client service from your computer in order to prevent a notable performance hit.

Disable Some Site Barriers, Blocks, Surveys, Ads, Banners, Etc.

August 23, 2015 / Brian Boll / 0 Comments

We have all experienced some element displayed on a webpage that didn’t quite sit well with us, whether it was an ad or some content wall requiring pay or survpay—payment in the form of survey completion (a fate often worse than actual payment). Thanks to browser developers keeping other developers in their thoughts, anyone who has updated their Firefox and Chrome browsers in the last half decade has access to this solution. So what I’m writing about here is manipulating HTML/XHTML/XML elements after they’re interpreted and rendered by your browser. But before I lose anyone, let me reassure that anyone can do this simply with a couple of right and left clicks in a matter of seconds with little understanding.

Note: Since deleting elements simply deletes them from your browser’s display to you, any changes can be easily reverted by reloading the webpage. No harm, no foul. Few developers check for deleted elements and I have never encountered any sort of user-punishment from the few that do. Keep in mind, any server-side logic such as requiring (as opposed to suggesting) login will not be able to be bypassed using this method if server-side logic is keeping you from accessing content.

DOM Structure

By now all browsers support the standard conventional representation of objects in HTML/XHTML/XML documents—the Document Object Model or DOM for short. So when you visit a webpage, your browser downloads the HTML, parses the HTML and organizes the elements that make up the document as nodes in a tree structure. The tie-in here is that you can delete these nodes and therefore delete them from your viewing. Nearly everything you view on a webpage is an element or series of elements, from the ‘Submit’ button to the annoying ad banner to the login form, all elements that get constructed as nodes.

Firefox and Chrome Element Inspector

Firefox and Chrome both have developer tools which can be accessed by right-clicking and selecting ‘Inspect Element’ or the hotkeys Ctrl+Shift+I. If you use the right-click option it will highlight the node corresponding to the element you clicked on. Simply hover over the highlighted text, right-click and select ‘Delete Node’. Below is an example of the navigation in Firefox, sharing a similar layout to Chrome. The tools can be closed by the ‘X’ I have highlighted or by using the hotkey again.

Me about to delete the node allowing me to write this post

Applying This To Blocks, Ads and Ethics

This will remain a rather simple but useful rudimentary solution that’s good to keep in your tool belt as a browser user. For instance it can also be used to save pictures that you are prevented from saving via right-click options menu.Hopefully I have explained this out in such a way that its uses require little imagination. I consider content walls the ultimate ‘gotcha’, bait and switch. It’s also rather quick and painless to do and is certainly a worthy way to closing any malicious pop-ups without having to worry about accidentally clicking on them.

While I understand the importance of ads and don’t condone closing them all, this is useful for closing annoying banner ads when streaming or otherwise recording your screen as well. And as far as anyone is concerned, you have to look at the ad to delete it so technically in the end they inevitably still win.

That being said, be sure to check out my upcoming post where I discuss blocking ads universally at their source using the hosts file as well as other solutions.

(Source: Wikipedia)

Parse Email Using PHP Scripting

August 21, 2015 / Brian Boll / 0 Comments

In a previous post, I described how to set up SMS notification for email. This solution had one large disadvantage out of the box, however: Emails with large bodies of text are sent to the phone as multiple SMS messages until it reaches a max SMS limit or the end of the email message. But it’s not ideal to think you’ll want to read entire email messages via texts, so instead I’m going to jump right into showing you how to format the message into more useful shorthand.

Note: Each email setup will be different depending on hosting situation but this post will largely deal with hosting where you have permissions to run scripts. Major hosts such as Yahoo, Gmail and Hotmail each have their own unique solution. I may post a Gmail solution in the near future using the Google App Scripts API.

PHPMailer

PHP has a mail() function but it’s awful compared to using a library, such as PHPMailer. PHPMailer provides an API for formatting outgoing emails which is especially useful to avoid having to construct email headers from scratch. It also provides more advanced features such as HTML emails, sending email attachments, and SMTP authentication. Plus it’s incredibly simple to install and can be downloaded here from the official github page (which also includes API documentation).

Install is as simple as extracting and uploading to your web disk in your home directory. Be sure to keep it out of your public_html directory.

Pipe Program

Now that PHPMailer is installed, let’s take a look at the script emailscrape.php:

#!/usr/bin/php -q
<?php 
  require_once('../PHPMailer_5.2.0/class.phpmailer.php'); 
  $credentials = require_once('../mail/mailconfig.php'); 
  //Read contents of email 1024 limit 
  $fd = fopen("php://stdin", "r"); 
  $email_content = "";
  while (!feof($fd)) { 
    $email_content .= fread($fd, 1024); 
  } 
  fclose($fd); 

  try { 
    $mail = new PHPMailer();                    // Initialize PHPMailer object
    $body = date("D j/n h:iA");                 // Grab timestamp
    $mail->IsSMTP();                            // Telling the class to use SMTP
    $mail->SMTPDebug = false;                   // SMTP debug information for testing
                                                // 1 = errors and messages
                                                // 2 = messages only
    $mail->SMTPAuth = true;                     // Enable SMTP authentication
    $mail->Host = $credentials['host'];         // Set the SMTP server
    $mail->Port = $credentials['port'];         // Set the SMTP port for the server
    $mail->Username = $credentials['username']; // SMTP account username
    $mail->Password = $credentials['password']; // SMTP account password
    $mail->IsHTML(false);                       // Not sending HTML email
    $mail->Body = $body;                        // Set message body to timestamp
    $to = $credentials['destination'];
    $mail->AddAddress($to);                     // Set destination email address
  
    //Initialize subject, default header to true and explode email by line
    $subject = "";
    $is_header= true;
    $lines = explode("\n", $email_content);
    //Check each line for Subject
    for ($i=0; $i < count($lines); $i++) {
      if ($is_header) {
        // Split out the subject portion
        if (preg_match("/^Subject: (.*)/", $lines[$i], $matches)) {
          $subject = $matches[1];
        }
        //Split out the sender information portion
        if (preg_match("/^From: (.*)/", $lines[$i], $matches)) {
          $from = $matches[1];
        }
      }
      if (trim($lines[$i])=="") {
      // empty line, header section has ended
        $is_header = false;
      }
    }
  $mail->Subject = $subject;
  $mail->setFrom($credentials['username'], $from);
  //Send mail and check for errors
  if($mail->Send()){
    //echo 'Email Successfully Sent!';
  } else {
    echo 'Email Sending Failed!';
  }
//Catch exceptions and output errors
} catch (phpmailerException $e) {
  echo $e->errorMessage();
} catch (Exception $e) {
  echo $e->getMessage();
}
?>

Program Breakdown

This script handles three functions:
1. Reads the email
2. Parses the email header info.
3. Constructs and sends our new formatted email

So any email piped into this script is read, the header fields ‘Subject’ and ‘From’ are found using regex matching and their contents are stripped, a new email is constructed that’s more SMS friendly and is forwarded on to the email recipient (in this case a phone via SMS).

There is one line that is especially important:

#!/usr/bin/php -q

This line is called a “shebang” and allows this script to be run as a shell command, the path defining the location of the compiled binary for the version of php that you want to interpret your script. This location should be default across hosting providers where PHP is installed. If you are having problems but have SSH access you can run the UNIX which command:

which php

to find the path to the executable for your shebang, otherwise contact your provider.

Upload Program To Web Server

Upload this script to your web disk outside of your public_html folder and with the chmod permissions set to 0700.
In my script I load in some of the more sensitive configuration details from a configuration file with the same permissions and also located elsewhere outside of my public_html folder.

Set Up Forwarding To Program

Now we rather simply need to set up the forwarding from our email address to the program. The address to forward from there is handled in the script (Note: If you already have a forwarder set up to forward directly to the phone’s email from following my previous post then delete that forwarder).

cPanel navigation

And there you have it!

Any emails received will now have a copy formatted and forwarded to your phone as soon as they’re received. I encourage you to experiment with various formatting options not just the timestamp example I used that fits my needs. Keep in mind that if there are any errors or other output from the script, a bounce back message will be sent to the original email sender. My advice is to test it thoroughly by sending yourself email from another email address and be sure to print errors to a log file rather than echoing them.

(Source: Bryce Glass via flickr)

Free & Reliable Email SMS Notification

August 7, 2015 / Brian Boll / 0 Comments

I switched to SMS notification because the normal mail client’s scheduled checks for notifications left a lot to be desired. What if an email arrived mere minutes after one check and didn’t notify me until the next scheduled check hours later? What if one of these emails was urgent and required a timely response? Why would I waste battery life on checking for mail when my phone already uses it to constantly check for SMS messages? Thoughts of my device having to constantly perform scheduled checks with mail servers is the kind of thing that kept me up at night. But no more!

Free Gmail SMS Alerts Discontinued

As of writing this Google has discontinued offering the service of SMS text alerts for email in any free capacity. However, even if you Google “Email SMS notifications” now you are still bombarded with an array of ever changing (read: outdated) hacks of their various free services. Of course, there are many services out there that will allow you this feature provided you pay, but what if I told you that you are already paying for this service?

An Unlikely Hero

Without further suspense I introduce the unlikely hero in our story: your major cell carrier. The cog in the machine for all of these services is an SMS gateway; this allows for the sending and receiving of texts through the mobile phone network. A premium service simply receives the email, parses the email into a more readable SMS format and then sends the text to the intended phone through their SMS gateway into the vast mobile network ether. You pay for premium services to format your emails into a more readable format ~~and use their gateways~~. But you are already paying to use your cell carrier’s SMS gateways and they have been quietly providing access to gateways that support Email-to-SMS conversion to anyone under contract.

Setup

Major cell carriers have most likely already set you up with a static email address linked to your phone number (not your device or any email client). Below is this wiki table of the major US-based cell carriers and their associated gateways:

US Carriers SMS Gateways

Discern Your Cell Phone’s Associated Email

You can easily discern your associated email using your phone number and the table above by following the simple format:
[10-digit phone number]@[carrier gateway domain]

Ex: If your phone number is 555-555-5555 on Verizon Wireless, your email is 5555555555@vtext.com

You should test your setup by sending an email to that address and promptly receiving an SMS text on the phone associated to that phone number.

Setting Up Email Forwarders

Now you only have to setup forwarding from your email of choice to the email your carrier provides your number. Depending on your email client this setup will differ but there are instructions to be found online for all major email clients. If your hosting provider uses cPanel, you can navigate to set up a forwarder like so:

cPanel Navigation

And with that, that’s it!

Any emails should now be forwarded to that middleman email and then converted automatically to SMS and sent to your phone from your carrier. Below are some more notes on this solution:

Stability and longevity from piggybacking on system of major cell providers

Easy to discern new email address and change forwarding when switching cell providers

Counts towards any SMS limits but no extra charges outside normal SMS charges

Long emails will be broken up into multiple 160-character texts until limit is reached (avg: 5)

No inherent formatting options, must receive “from”, “subject” and “message” in entirety*

*Check my post here for a guide to formatting the SMS notifications to your liking with a little scripting know-how.

Theme by Anders Norén — Up ↑