Wednesday, November 09, 2011

Fun with Applescript, VPN and Proxy

I have a lot of digital pictures and home movies, and I'm supposed to be smart enough not to lose them by mistake or hardware failure. So part of my backup process is to backup files off-site across the internet using Crashplan.

This worked fine at first, but over time something happened and my computers just couldn't see each other unless I connected over a VPN. Bringing up the VPN is a manual step (enough of them and you won't have backups anymore), and so is restarting it when it drops, which happens at least once an hour.

If I don't have too much data, then I can start the VPN, nudge Crashplan to notice it right away, and get my files sent off-site. But if there is a lot, home broadband upload speed is slow, and not much will be done before the VPN gets knocked down. Then the backup will stop until I come back and fix it, which probably won't happen. Usually when I see that I have hundreds of megs or more of stuff not finishing, I will just bring my computer to the other site, plug it into gigabit ethernet LAN where it can finish pretty quickly.

After staying up too late on the internet the other night, I found out how to automatically restart the VPN when it falls down. So now I can just leave the computer on at home and bigger backups get farther without babysitting.

It's just an Applescript, created with "Applescript Editor" that comes with every Mac, "Saved As" format "Application," with the checkbox for "Stay Open" checked. I saved it under my Home directory in ~/Applications/restartVPN.app

on idle
    tell application "System Events"
        tell current location of network preferences
            set myConnection to the service "WHATEVER THE SERVICENAME IS"
            if current configuration of myConnection is not connected then
                connect myConnection
            end if
        end tell
        return 60
    end tell
end idle

The "WHATEVER THE SERVICENAME IS" is whatever the VPN service is called in the list of services under System Preferences, Network. If it's a long name, it might be shortened in the list with an ellipsis but you can see the full name by either hitting the "Advanced" button and looking at the top of the next pane, or by checking the "Show VPN status in menubar," then clicking that menubar icon.

Under Lion, if you need to add routes for the VPN, you put them in /etc/ppp/ip-up


#!/bin/sh
if [[ "$5" == "123.456.789.253" ]]; then
/sbin/route add -net 10.10.10.0/24 123.456.789.253
fi

Now whenever I start the VPN connection, I also invoke Spotlight (Command+Spacebar), start typing "restartVPN.app" and after a couple of letters, when that result jumps to the top of the list, hit the Enter key and the will app run until I quit it. With VPN disconnections of less than 60 seconds, Crashplan can keep uploading to the other side for as long as the machine is on.

Hungry for more Applescript, I thought about how many manual steps there are to setup system wide proxying through an ssh tunnel. First, open Terminal, run "ssh -D 9999 me@somewhereelse" to connect to the other host (using ssh keys) where I want my tunneled traffic to come out. Then System Preferences, Network, click my current servicename (typically Wi-Fi), Advanced, Proxies, then enable checkbox for "SOCKS proxy" ("SOCKS proxy server" on that pane should have "localhost:9999"), then "OK" and then finally "Apply."

What a hassle, but then, SOCKS aware apps like Firefox will automatically send their traffic through the tunnel, so my IP address while browsing will be the IP of the place I ssh'd to, not the IP assigned by my Internet Service Provider at home. However, not all programs are SOCKS aware. Transmission, uTorrent and wget are not. When they connect to places on the net, they do it with your real, not proxied, IP. Safari, Vuze and Xtorrent are: TCP only reveals your proxy IP to those you connect to. Curl can do SOCKS, but you have to ask it special, and even when the tunnel is down, it still acts like it is there, which was confusing to me and I didn't spend much time trying to figure it out. 

So, to automate setting and unsetting the system wide SOCKS proxy,  here's a bash shell script. It scans to see if it's already on or not. If it's on, it turns it off. If it's off, it turns it on, kills any ssh process that looks like a leftover from an old tunnel, creates a new ssh tunnel, and prints the current status and my IP address as seen from the other side of any tunnel in a Growl notification.

The script uses "osacript" in order to run some Applescript to launch Safari and read a webpage to figure out the IP address I look like on the internet. The Applescript is because Safari will use the tunnel if it is there -- wget cannot (and curl was just weird).

#!/bin/bash

device="Wi-Fi"
#device="Ethernet"

function myip {
osascript <<
EOF

property myURL : "http://automation.whatismyip.com/n09230945.asp"

tell application "Safari"
 

    if (count documents) = 0 then
        make new document with properties {URL:myURL}
    else
        set URL of document 1 to myURL
    end if 



    launch
    repeat until exists (window 1)
    end repeat

    repeat with w in (get every window)
    set miniaturized of w to true
    end repeat

    tell window 1
        delay 1
        set mySrc to source of the current tab
        return mySrc
    end tell

end tell
EOF
}

if [[ `scutil --proxy | grep SOCKSEnable | awk '{ print $3 }'` == "1" ]]; then
    networksetup -setsocksfirewallproxystate "$device" off
    proxyState="disabled"
else
    networksetup -setsocksfirewallproxy "$device" 127.0.0.1 9999 off
    kill `lsof -i 4TCP@localhost:9999 -P -sTCP:LISTEN -a -c /^ssh$/ | awk '{ print $2 }' | tail -1`; 2> /dev/null
    ssh -N -D 9999 me@somewhereelse &
    proxyState="enabled"
    sleep 5
fi

if [[ -e /usr/local/bin/growlnotify ]]; then
       /usr/local/bin/growlnotify -m "IP: `myip`." "SOCKS Proxy: $proxyState"
else
    echo "SOCKS Proxy $proxyState. IP: `myip`."
fi


This shell script could just be run from a Terminal window, but I decided to turn it into an "application" with Automator.app so that I could invoke it with a Quicksilver keyboard shortcut. That is Command+Spacebar, type "automator" (click it or hit Enter when "Automator" jumps to the top), then Command+N, click "Application" and "Choose" from the "Choose a type" dialogue, then drag the "Run Shell Script" action into the big empty workflow area, erase the default "cat" text from the input box and replace it with the full path to where you saved the shell script, like /Users/whoeveryouare/togglesox.sh then "File", "Save" as format "application." I saved mine as  togglesox.app under the Applications directory below my home directory.

Then go get really frustrated trying to remember how to set a Trigger for a keystroke combination in Quicksilver while Lion acts buggy and freezes Quicksilver or makes it disappear. The keystroke should "open" /Users/whoeveryouare/togglesox.app or wherever you saved it. Alfred or regular OSX Keyboard Shortcuts could also launch it.

So now I can do Command+Shift+p and a little Growl notice will popup on my screen telling me "Proxy Enabled; IP address [other side of my tunnel]." Do it again and Growl shows "Proxy Disabled; IP address [given by my ISP ]."

I also found http://checkmytorrentip.com/ to be helpful in telling you whether your torrent client is going through your tunnel or not, since I just found out that a checkbox setting I enabled months ago in Transmission preferences for something about "SOCKS proxy" only referred to trackers, not to peers, and that setting is now gone and feature removed in the current version.

All this took way more time to get working than could possibly be justified, so please make some use of it.

Monday, April 04, 2011

Stop form input from capturing/ignoring certain keypresses

Too many web forms have some stupid javascript that tries to limit what keys you can press while you are focused in a certain input element. For example, in my bank billpay system, they have javascript that makes it so you can only type numbers while you are in the "Zip Code" field. When they do that, it disables me from using the CMD+v to paste a zip code in the field, because "CMD+v" is not a number.

Here is a sample of the javascript they use:

function numbersonly(myfield, e)
{
    var key, keychar;
    if (e) key = e.which;
    else return true;
   
    // allow control keys
    if ((key==null) || (key==0) || (key==8) ||
         (key==9) || (key==13) || (key==27) )
        return true;

    // numbers
    else if ((("0123456789").indexOf(keychar) > -1))  return true;
    else return false;
}

Then they have HTML like this:

<input onKeyPress="return numbersonly(this, event)" type="text" />

So, if you are a Windows user, you probably are allowed to CONTROL+v to paste a zip code. But Mac users just see the "Edit" menu flicker for a moment and their paste command (keycode 118) is dropped on the floor. Neither can you "CMD+," (keycode 44) to get Application Preferences, or use the metakey to do any other Firefox action while in that form field. It should validate only when the user is ready to submit, not while typing.

Install Greasemonkey and rip that crap out with this script:

// ==UserScript==
// @name           Get Off My Keypress
// @namespace      http://gomk.amulder.modwest.com/gomk.user.js
// @description    stop websites from setting event handlers to capture your keypresses
// @include        https://sitethatbothersyou.com/*
// @require  http://ajax.googleapis.com/ajax/libs/jquery/1.5.1/jquery.min.js
// ==/UserScript==

$(document).ready(function() {
    // unsafeWindow.console.log("testing firebug after jq");     
    $("input").each(function(i, elem) {
        // unsafeWindow.console.log(i + $(elem).attr("name"));
        var h = elem.getAttribute("onKeyPress");
        // might return "return numbersonly(this, event)"
        if(h) elem.setAttribute("onKeyPress", "return true");
    });     
}); 

Having to use Greasemonkey is really more complicated than it should be. We should be able to use the new CAPS (capabilities) security policy settings built into Firefox 4 to grant noAccess to specific websites to set onkeypress events or read what keys you are pushing on. I tried several iterations using the "Control de Scripts" extension to block these:

HTMLInputElement.onKeyPress
Window.numbersonly
Window.onKeyPress
Window.onkeypress
event.preventDefault 

but none worked. I think this is because I don't really grasp what an event handler is or where it is in the DOM. Maybe someone else can comment to get a solution that is native to the browser, without needing a whole extension just to do this one simple thing.

Saturday, March 26, 2011

Review of OWC 480G SSD laptop drive

After waiting for SSD notebook drives to get big enough that I could fit everything on it that I carry around on my Hitachi 500G 7200 rpm mechanical drive, OWC finally came out with a 480G SSD last summer that was over a thousand dollars. Just recently, they lowered the price a lot and I got mine for a total of $908, including Fedex 2 Day shipping and a $25 rebate for using Amazon Checkout when I bought it through their website.

I think the price change is related to recent Sandforce chip changes and maybe also that the next rev Sandforce 2000, reportedly twice as fast as these, may be only 8 weeks away.

I got the drive on time from OWC and, unlike the Seagate that just came in an antistatic bag like a McDonald hamburger, this one came in nice retail packaging. Since I mention the Seagate, I should also say that this Hitachi that I eventually replaced it with was slightly slower, but without the confidence eroding clicking, and maybe less pinwheels until recently.

Before installing the new SSD drive, I ran some crude benchmark tests, then used SuperDuper! to clone my old internal drive to my external backup, a firewire 800 G-Drive Mini, which are the nicest bus-powered enclosures I have seen. After making a bootable backup, I opened the laptop to install the new SSD drive, then booted from the firewire backup.

Upon logging in, OSX offered to "initialize" the unrecognized/unformatted internal SSD drive. When I said yes, it opened Disk Utility, where I clicked the disk, named it, and erased/formated it as MacOS Extended (Journaled). Then I launched SuperDuper! and told it to restore everything from the external backup drive that I had booted from onto the empty internal SSD. That process read data off the 5400rpm backup drive and over the firewire at about 50MB/s (according to the "Disk Activity" graph in Activity Monitor). While 300 gigs went from here to there I did laundry and played with my baby.

Once my backup was restored to the internal SSD, I rebooted and here is the comparison:

Boot time comparison
7200rpm OWC SSD
Apple logo 38 sec (but probably with unset startup drive system preference setting) 4 sec (after setting startup drive in system preferences)
Login Window +30 sec +13 sec
Boot Total 68 sec 17 sec
Login and launch all startup items, including Firefox +80 sec +9 sec
Usable Total 148 sec (2.5 minutes) 26 seconds

I was pretty impressed with how fast it launched all my stuff after I logged in. The OWC website linked above has a graph showing powerup to desktop in 19 seconds. Mine's not doing that, but I am still happy with it.  Correction on March 28,2011: Thanks to the article on macperformanceblog that tells you to set your startup disk after upgrading a hard drive, my machine really does go from off to login window in < 20 seconds! If I can type my password fast enough, my machine is totally ready to use in a total of < 35 seconds!

Here are 2 more before/after comparisons.

Xbench

dd test
Writing to /test on the 7200 rpm drive:
#sudo time dd if=/dev/zero of=/Volumes/Macintosh\ HD/test bs=1024k
[waited a while, then CNTRL+c]
97679048704 bytes transferred in 1565.832047 secs (62381562 bytes/sec)

Reading /test on the 7200 rpm drive:
#sudo time dd of=/dev/null if=/Volumes/Macintosh\ HD/test bs=1024k
[waited a while, then CNTRL+C]
39557529600 bytes transferred in 543.910842 secs (72727967 bytes/sec)
#sudo rm /test

Writing to /test on the SSD:
#sudo time dd if=/dev/zero of=/Volumes/Macintosh\ SSD/test bs=1024k count=16384;
17179869184 bytes transferred in 66.070450 secs (260023493 bytes/sec)

Reading /test on the SSD:
#sudo time dd of=/dev/null if=/Volumes/Macintosh\ SSD/test bs=1024k
17179869184 bytes transferred in 61.126414 secs (281054753 bytes/sec)
#sudo rm /test

So that tells me compared to the 7200rpm Hitachi writing 59MB/s and reading 69MB/s, the SSD's 248MB/s writes and 268MB/s reads are about 4 times faster. Plus, the battery runtime that the battery monitor is reporting now looks about an hour longer than it would have been with the mechanical drive.

This and the cheap 8 Gig RAM upgrade also from OWC put an end to any other hardware upgrades for this computer.

Wednesday, January 20, 2010

BoA Nickname Payees Greasemonkey Script

Bank of America does not let you create "nicknames" for your payees on its Quickpay online billpay webpage. This means that if you have more than 1 account with the same entity, or different individuals with the same bank, it is very hard to tell your payees apart when you go to make a payment. This can result you sending a payment to the wrong payee.

You may end up with identical looking payees if you pay utility bills to the same company for more than 1 address, or if you frequently send money to family members who use the same bank as each other. In these cases, the way BoA's Quickpay page is now, you would have to memorize which account number goes with which payee in order to tell them apart and make a payment to the right person. See the screenshot below for how confusing it is:



In the screenshot above, the first "BANK OF AMERICA CHECKING" in the payee list is for one person, and the second one is for someone else, who also happens to bank at BoA. I wrote this Greasemonkey User Script in order to let people who use the BoA billpay create useful nicknames for these payees, since the BoA system doesn't allow it.

After you install the Greasemonkey script, you'll be able to create and edit nicknames for your payees by clicking the link to "create nickname" or by clicking the nickname to edit if you already created one. All other items display on the QuickPay page as usual. After you've made some nicknames, the billpay screen will look like this:



Now you can tell your payees apart without having to remember which account number is which.

Here is the script:


You can also download it from https://gist.github.com/3272964

Sunday, January 03, 2010

Thinking about using itshidden.com

I was thinking of using the anonymous VPN service from itshidden.com. I was even thinking of paying for it, despite not knowing the reliability or trustworthiness of those on the other end. So I signed up and got this:



So, just nevermind.

Tuesday, May 19, 2009

Review of new Seagate 500g 7200rpm laptop hard drive

When my 1 year old Hitachi laptop hard drive died in March, I started shopping for a bigger one. That's when I found the new Seagate Momentus 7200.4 model ST9500420AS with 500 gigs of space at 7200rpm.

At that time though, the drive was out of stock everywhere as Seagate seemed to have stopped production to fix some engineering problems. Since even now (May 2009) this is the biggest and fastest consumer 2.5 inch drive available, I decided to wait for it.

In the mean time, I read reviews on it, which were mixed. Some people who got their hands on one of the first run models reported them slow, noisy, buggy and hot. Other owners reported back that the drives were fine. Two more good reviews issued from barefeats and hardwarelogic.

After about 2 months of waiting, these drives were for sale once again, this time with updated firmware 2SDM1. This is the one I bought online from WiredZone for $133 with free shipping. It's just a brown box, egg foam and anti-static bag. No glossy fanfare, manuals or any of that. I don't know if that's typical.



Before evicting the old (warranty replacement) Hitachi 200g 7200rpm, I decided to take some crude benchmarks of it so I'd know whether the new Seagate was any better. Say goodbye to HTS722020K9SA00 Made in Thailand:



Hello Seagate Hecho in China:

Seagate ST9500420AS 7200rpm 500gig, firmware: 2SDM1

After copying a bootable backup onto an external firewire drive with the excellent SuperDuper! cloning software, I was ready to swap drives. Disassembly instructions for the model 3,1 MacBook Pro are at iFixit. If you have a Bugs Bunny video, then you won't have to unhook the delicate ribbon cable that connects the keyboard to the motherboard.



After the swap, I booted from the external firewire drive and used SuperDuper! to clone back onto the new empty internal hard drive. Then a reboot and everything is running on the new Seagate.

Now, for the comparisons, which may not be quite fair because the Hitachi was almost full for the benchmarks, while the Seagate was mostly empty.

Boot Time
HitachiSeagate
Apple Logo: 51 sec
Login Window: +33 sec
Total: 84 secs
Apple Logo: 16 sec
Login Window: +40 sec
Total: 56 secs


Xbench
HitachiSeagate
Results 43.50
System Info
Xbench Version 1.3
System Version 10.5.6 (9G55)
Physical RAM 4096 MB
Model MacBookPro3,1
Drive Type Hitachi HTS722020K9SA00
Disk Test 43.50
Sequential 79.03
Uncached Write 108.92 66.88 MB/sec [4K blocks]
Uncached Write 110.86 62.73 MB/sec [256K blocks]
Uncached Read 40.92 11.98 MB/sec [4K blocks]
Uncached Read 125.34 62.99 MB/sec [256K blocks]
Random 30.01
Uncached Write 9.76 1.03 MB/sec [4K blocks]
Uncached Write 97.19 31.11 MB/sec [256K blocks]
Uncached Read 77.83 0.55 MB/sec [4K blocks]
Uncached Read 130.73 24.26 MB/sec [256K blocks]
Results 52.73
System Info
Xbench Version 1.3
System Version 10.5.6 (9G55)
Physical RAM 4096 MB
Model MacBookPro3,1
Drive Type ST9500420AS
Disk Test 52.73
Sequential 119.09
Uncached Write 164.42 100.95 MB/sec [4K blocks]
Uncached Write 146.68 82.99 MB/sec [256K blocks]
Uncached Read 65.58 19.19 MB/sec [4K blocks]
Uncached Read 183.84 92.39 MB/sec [256K blocks]
Random 33.86
Uncached Write 10.73 1.14 MB/sec [4K blocks]
Uncached Write 171.66 54.96 MB/sec [256K blocks]
Uncached Read 80.56 0.57 MB/sec [4K blocks]
Uncached Read 148.96 27.64 MB/sec [256K blocks]


Bonnie++
Hitachi:
Version 1.93c       ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
arf.local 16G 297 97 52023 18 24538 9 332 95 55015 11 105.9 10
Latency 80374us 687ms 610ms 121ms 213ms 4029ms
Version 1.93c ------Sequential Create------ --------Random Create--------
arf.local -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 6111 55 +++++ +++ 9257 61 367 8 +++++ +++ 151 4
Latency 76653us 920us 79744us 333ms 1399us 414ms

Seagate:
Version 1.93c       ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
arf.local 16G 293 96 90295 32 36049 14 343 98 91548 20 164.4 13
Latency 134ms 388ms 217ms 85474us 132ms 1942ms
Version 1.93c ------Sequential Create------ --------Random Create--------
arf.local -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 5313 56 +++++ +++ 5567 41 320 8 +++++ +++ 149 5
Latency 106ms 200us 151ms 497ms 380us 322ms


dd
I ran the write in root "/" directory (and therefore had to use sudo) to avoid writing in my home directory, which is Filevault and would probably skew performance downward. The 1st dd in each section below is writing, the 2nd is reading.

HitachiSeagate
$ sudo time dd if=/dev/zero of=/Volumes/Macintosh\ HD/test bs=1024k count=16384;
17179869184 bytes transferred in 319.400409 secs (53787875 bytes/sec)

$ time dd of=/dev/null if=/Volumes/Macintosh\ HD/test bs=1024k
17179869184 bytes transferred in 305.974147 secs (56148107 bytes/sec)
$ sudo time dd if=/dev/zero of=/Volumes/Macintosh\ HD/test bs=1024k
17179869184 bytes transferred in 188.208635 secs (91280983 bytes/sec)

$ time dd of=/dev/null if=/Volumes/Macintosh\ HD/test bs=1024k
17179869184 bytes transferred in 180.531769 secs (95162581 bytes/sec)


The result of the tests showed that in addition to more than doubling my disk space with the new drive, it is also objectively faster than the old one, even at the same spindle speeds. However, this is probably just because the new one is mostly empty and the old one was mostly full. Performance will always be a lot better when data is on the beginning instead of the end of a mechanical drive.

The last bit of info to share is from SMART, accessed with smartctl from smartmontools. The Hitachi was in pretty good shape, aside from the strange value for Power-Off_Retract_Count. Don;t have a clue what that one means:

Hitachi SMART
Model Family:     Hitachi Travelstar 7K200
Device Model: Hitachi HTS722020K9SA00
Serial Number: 080830DP0470DTGP3MMC
Firmware Version: DC4AC77A
User Capacity: 200,049,647,616 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 3f
Local Time is: Sun May 17 23:31:07 2009 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled


SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 062 Pre-fail Always - 0
2 Throughput_Performance 0x0005 100 100 040 Pre-fail Offline - 0
3 Spin_Up_Time 0x0007 176 176 033 Pre-fail Always - 1
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 200
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 100 100 040 Pre-fail Offline - 0
9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 342
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 192
191 G-Sense_Error_Rate 0x000a 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 42954326020
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 8991
194 Temperature_Celsius 0x0002 130 130 000 Old_age Always - 42 (Lifetime Min/Max 14/47)
195 Hardware_ECC_Recovered 0x000a 100 100 000 Old_age Always - 0
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0
223 Load_Retry_Count 0x000a 100 100 000 Old_age Always - 0


The Seagate, on the otherhand, looked alarming when I first checked it out. I thought the drive was defective and was ready to send it back for RMA:

Seagate SMART
$ sudo smartctl -s on /dev/disk0
SMART Enabled.

$ sudo smartctl -a /dev/disk0
Device Model: ST9500420AS
Serial Number: 5VJ079ZE
Firmware Version: 0002SDM1
User Capacity: 500,107,862,016 bytes
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 4
Local Time is: Wed May 20 00:42:19 2009 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
...
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 118 100 006 Pre-fail Always - 184273167
3 Spin_Up_Time 0x0003 100 100 085 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 2
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 100 253 030 Pre-fail Always - 139058
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 24
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 037 020 Old_age Always - 5
184 Unknown_Attribute 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 062 052 045 Old_age Always - 38 (Lifetime Min/Max 28/41)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 0
193 Load_Cycle_Count 0x0032 099 099 000 Old_age Always - 3798
194 Temperature_Celsius 0x0022 038 048 000 Old_age Always - 38 (0 22 0 0)
195 Hardware_ECC_Recovered 0x001a 045 045 000 Old_age Always - 184273167
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 70690866724884
241 Unknown_Attribute 0x0000 100 253 000 Old_age Offline - 2076453070
242 Unknown_Attribute 0x0000 100 253 000 Old_age Offline - 2307582568
254 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 21 -
# 2 Extended offline Aborted by host 90% 21 -
# 3 Extended offline Aborted by host 60% 9 -
# 4 Extended offline Aborted by host 50% 2 -


Raw_Read_Error_Rate, Seek_Error_Rate and Hardware_ECC_Recovered make it look like the disk is dying. Attributes 240, 241, 242 are nonsensical. After investigating though, it seems that these type of values on those attributes are just normal for a Seagate.

A few searches on these attributes will find many discussions where people are concluding that these odd values don't indicate a problem. Seagate also has a KB article basically warning users not to pay attention to those SMART values. Apparently they stuff their own proprietary values into the SMART circuits which only becomes meaningful when pulled through their "Seatools" disk analyzer software. Regular SMART software tools that follow the published SMART protocols won't be able to make any use of Seagate's stored raw values for those attributes. Incidentally, there is no Mac version of Seatools.

I've had my new Momentus drive running now for about 24 hours. Over the last 4 hours while I've been using the machine, I have heard a pretty loud "CLUNK" twice. Probably the heads parking or unparking for powersaving mode. The Hitachi never did that, but it's only happened twice and other than that, I can't tell the difference between this drive and the old one by noise, vibration or temperature. Tests indicate that there are no errors that other owners were complaining about this past winter with the older firmware rev, and it is a bit faster than Hecho in Thailand.

I guess it's a good one but I'll still keep up the SuperDuper! onsite and Crashplan offsite regimen.

Monday, December 08, 2008

use jQuery AJAX to create options in a dropdown menu

If you love to hate Javascript, that half-breed, amateur-magnet language, and every onload() and eval() you ever saw, then jQuery just rained on your parade. Now, with jQuery, so many things are elegant and easy - the opposite of everything you've ever known about Javascript.

Here's how to populate a select menu's option list (values and labels) with data retrieved from an AJAX request. The impatient may jump to a working demo to see if this is even what you want. Maybe you were searching for some sink cleaning product.

Materials:
  • 1 webpage that loads jQuery and makes an AJAX request (page.html)
  • 1 script that receives the AJAX request and answers it (script.php)
Unlike so many pages written in plain Javascript, which don't function or are missing content in browsers that do not execute Javascript (Lynx, Googlebot, people who dont want your crappy code heating up their CPU and turned it off in Preferences), jQuery's philosophy is that if your browser will execute Javascript, then the page will be better, but if not, then the page should "degrade gracefully" and still at least mostly work. So in this tutorial, the demo page is served with an initial select menu that should be good enough, in case the user doesn't run the script.

Here's the markup that we start with (page.html): A form with an input field, select menu and a button that will trigger our script. In this example, a user enters a zip code in the input box and clicks the button. That will trigger an AJAX request to another resource, sending the zip code and retrieving a bunch of shipping options, which will then magically fill the dropdown menu.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>jQuery ajax demo populating select dropdown menu</title>
</head>
<body>
<form>
Zip: <input name="zip" type="text" size="5" maxlength="10" id="zip" value="" /><br />
<select name='shipping_method' id="shipping_method">
<option value="0" selected="selected">Select Shipping Method</option>
<option value='FEDEX_2_DAY' >FedEx 2 Day</option>
<option value='FEDEX_EXPRESS_SAVER' >FedEx 3 Day</option>
<option value='INTERNATIONAL_PRIORITY' >FedEx International</option>
<option value='PRIORITY_OVERNIGHT' >FedEx Priority Overnight</option>
<option value='STANDARD_OVERNIGHT' >FedEx Standard Overnight</option>
</select>
<input id="getrates" type="button" value="Lookup Shipping Rates" /><br />
</form>
</body>
</html>


Notice that there are no Javascript functions strewn into the markup as tag attributes (no onclick(), no onmouseover()). That's because jQuery separates code for behavior from code for presentation. All the jQuery code will go in the "head." From high up there, it can hook into the DOM using only its patent pending "selectors."

So now let me show you what to add inside the "head" tag in order to load jQuery on the page, and then jQuery code to write that will do all the work.

First, a tangent: Normally, you would download the jQuery.js libraries from jQuery.com onto your own webserver, and serve them to your visitors from there with a "script src" tag. That's fine if you want to do it that way, but there's another option to direct visitors get those libs from Google instead. There are a lot of good reasons for offloading this job, so I leave it to you to read about it. In this example, that's what we're doing.

So, first, add this inside your head tag (before or after title tag) to get your visitor to load the jQuery libraries:

<script src="http://www.google.com/jsapi"></script>
<script type="text/javascript">
google.load("jquery", "1");
google.load("jqueryui", "1.5.2");
</script>


Now the browser understands jQuery. All the rest of your jQuery code can come next, in separate script tags, still in head:

<script type="text/javascript">
// When the DOM is ready to have events hook into it...
$(document).ready(function() {

// when the DOM element with id="getrates" is clicked....
$("#getrates").click(function() {

// make that button you clicked disappear...
$(this).hide(); // opposite of show() // See jQueryUI for more info

// and issue an AJAX request to a PHP script in the same directory
// getJSON is a method that expects a JSON-encoded data structure to be returned
// there are other AJAX methods too. See http://docs.jquery.com/Ajax
$.getJSON("script.php", // 1st arg to getJSON is the URI of the script
{
zipcode: $("#zip").val(),
random: "noise"
},
// 2nd arg to getJSON is an array of key-value pairs to send
// to the script. As many as you want.
// left-side is the GET variable name as seen by the target
// script, right-side is the value that will be sent.
// the above 2 args will cause the AJAX script to be hit with
// the query string: ?zipcode=90019&random=noise
// assuming that the user typed "90019" into the
// element on this page with the id="zip"


// 3rd arg is the callback function for the AJAX response
// The script.php responds in a JSON format so jQuery can
// understand the data structure natively, without you
// writing awful parsing of your own:
function(j) {
// erase all OPTIONs from existing select menu on the page
$('#shipping_method options').remove();

// You will rebuild new options based on the JSON response...
var options = '<option value="">Choose Shipping Method</option>';
// "j" is the json object that was output by your PHP script
// it is the array of key-value pairs to turn
// into option value/labels...
for (var i = 0; i < j.length; i++)
{
options += '<option value="' +
j[i].optionValue + '">' +
j[i].optionDisplay +
'</option>';
}
// stick these new options in the existing select menu
$("#shipping_method").html(options);
// now your select menu is rebuilt with dynamic info
}
); // end getJSON
}); // end clicked button to trigger AJAX
}); // end document ready
</script>


That's it. When the user clicks the button, an AJAX request is sent to script.php, and the response is used to rebuild the shipping_method dropdown menu. Prices appear inside the options list before your very eyes. If you want to see a working demo, check here

You may also want to copy this, name it "script.php" and save it on your server in the same directory as the html page above. It outputs a canned JSON answer that the jQuery code will use to make the select options.

<?php
# script.php

$pretend_results = array('PRIORITY_OVERNIGHT' => 39.69,
'STANDARD_OVERNIGHT' => 48.45,
'FEDEX_2_DAY' => 19.75,
'FEDEX_EXPRESS_SAVER' => 15.75);

$haOptions = array();
foreach($pretend_results as $method => $cost)
{
$haOptions[] = array('optionValue' => $method, 'optionDisplay' => "$method $$cost");
}

# make JSON object that will populate select dropdown menu options

if(function_exists('json_encode'))
{
echo json_encode($haOptions); # this puts the php array in the funny javascript array/object
# format so you don't have to know how to translate manually
}

else
{
# some lame web hosts dont have a new version of PHP (5.2+) that includes json functions in core
# so here, I fake it for you so you will have a working demo:
echo '[{"optionValue":"PRIORITY_OVERNIGHT","optionDisplay":"PRIORITY_OVERNIGHT $39.69"},{"optionValue":"STANDARD_OVERNIGHT","optionDisplay":"STANDARD_OVERNIGHT $48.45"},{"optionValue":"FEDEX_2_DAY","optionDisplay":"FEDEX_2_DAY $19.75"},{"optionValue":"FEDEX_EXPRESS_SAVER","optionDisplay":"FEDEX_EXPRESS_SAVER $15.75"}]';
}

# this output is what the jQuery ajax request will receive and parse in its ajax callback function
exit;
?>


When pasting the above PHP script into your editor, if your server does not have the json_encode() function (PHP version < 5.2) then be careful to NOT let your editor (like pico) wrap the long line dummy JSON string with hard line breaks. Hard returns in that data structure will break the fragile thing and your AJAX callback function will not execute.

It is still Javascript, after all, what did you expect?

Sunday, March 02, 2008

Reasonable Backups of Filevault

It doesn't take much web searching to come to the conclusion that the new Time Machine in MacOS 10.5 does not work well with Filevault.

The problem is that to Time Machine, a home directory protected with Filevault is just one big "sparse image" encrypted file. Although it will happily backup this file, doing that defeats one of the purposes of TM, which is to give you snapshots of every individual file from different times, so that you can go back through them and preview them easily before restoring.

If TM is backing up this giant disk image each time, then it is spending all your disk space on your backup drive on the whole disk image for every snapshot. This is a total waste of space. Without Filevault, the behavior would be to take a snapshot only of the changed files, so that your backup drive was only using space to store 1 copy of your files, plus the changes for each snapshot. Another problem with the interaction between FV and TM out of the box is that it's not very convenient in the "Cover Flow" interface to browse through the encrypted images, nor to have to provide a passphrase for each and mount each in order to look at the files inside.

Therefore, I decided to use "rsync" (from Terminal) to backup my home directory to an external drive while I am logged in. In order to keep my files secure on the backup drive, I decided to encrypt that whole device with Truecrypt, which just recently added support for MacOSX.

First I downloaded the Truecrypt .dmg file, mounted that by doubleclicking it, then ran the Truecrypt installer inside there. Once Truecrypt was installed on the Mac, I ran it and told it to encrypt the whole external USB backup drive.

After that was finished, I mounted the new volume according to the "Beginner Tutorial" in the TC documentation. At this time, TC could only create the volume as a FAT filesystem. Because I've been burned before by FAT's maximum filesize of 4G (tarring some stuff directly to the backup drive and having my tarball silently truncated at 4g), I wanted a real filesystem for my backups.

To change the filesystem of the mounted Truecrypt volume, I opened Disk Utilities from the Applications, Utilities menu in the Finder and, while Truecrypt volume is still mounted (so you see it without the encryption), told it to partition the new volume 200G HFS+ and 50G FAT. I left a FAT partition on it so that I could still use the drive on other non-Mac computers.

After the TC volume was re-partitioned and reformated, I was ready to run rsync to copy my home directory in there:
rsync --archive --progress --verbose \
--exclude '.Spotlight-V100' --exclude '.fseventsd' \
--exclude 'Desktop ' --exclude 'Library/*' \
--exclude 'Downloads/*' --exclude 'Music/*'
--exclude 'Public/*' --exclude 'Sites/*'
~me /Volumes/MacBackup/backup
Where my username on the Mac is "me" and the HFS partition inside the Truecrypt volume is "MacBackup" and the directory inside there where I want all my backup stuff is "backup." The result of the command is that everything in my home directory, including hidden files that begin with a '.' like .bashrc, will be copied to the backup directory -- except for a few subdirs of the home that I don't care about and have excluded.

While figuring out which rsync command will work for you, add the "--dry-run " option in until you get it right. I will be saving that command in a shell script that I will periodically execute after connecting the USB drive and running Truecrypt to unlock and mount it.

The reason that I am involving Truecrypt at all is just so that I could use the backup drive on other non-Mac machines, since it is cross platform Windows/Linux/Mac encryption. If I didn't care about the cross platform stuff, I would just have used Apple's Disk Utilities to create an encrypted disk image on the USB drive, and stored backups in there. I sort of defeated some of that purpose by using an Apple-only HFS partition, but maybe in the future there will be a better cross platform filesystem to select from the Disk Utility menu that will also support files larger than 4G.

Friday, November 16, 2007

Interested in GPL code for your closed source project?

This post does not really fit the theme of this blog as a "fix" but may may help someone avoid broken-ness in the first place.

Question: If I am writing closed source PHP software intended for distribution (not just for use as a web service, see http://radar.oreilly.com/archives/2007/07/the_gpl_and_sof_1.html), and I incorporate a few GPL components, does my software become "infected" and necessarily GPL as well?

Answer:
The answer to this question would matter to someone who has invested, or is about to invest, significant resources in what he/she may consider an original work, but who may also be tempted to incorporate freely available GPL'd software in the process.

From information you can gather from the Free Software Foundation, the answer seems simple:
"[P]eople have been wondering what the rules are when you link to some GPLv3-covered code. They're the same as they were under GPLv2: the combined work you create needs to be GPLed as well."
(from http://www.fsf.org/blogs/licensing/2007-10-18-gplv3-fud)

or from Richard Stallman:
"I once found out about a non-free program which was designed to use Readline [a library covered by GPL], and told the developer this was not allowed. He could have taken command-line editing out of the program, but what he actually did was rerelease it under the GPL."
(from http://www.gnu.org/philosophy/pragmatic.html)

and even more directly from GNU:
"You cannot incorporate GPL-covered software in a proprietary system.... A system incorporating a GPL-covered program is an extended version of that program.... [and] must be released under the GPL."
(from http://www.gnu.org/licenses/gpl-faq.html#GPLInProprietarySystem)

Despite these forceful admonitions that if A, a GPL work, is combined with B, then the resulting A+B then has to be GPL, there are places where even GNU concedes that it is not always the case that when one piece of GPL software is distributed with some other software, that the GPL will always override whatever "some other's" redistribution license might have been.

For example, the Linux kernel is GPL2 only (not "or any later version") and might never be changed by the kernel developers to GPL3 (http://radar.oreilly.com/archives/2007/04/gplv3_linux_and.html), while many other GNU programs in the GNU/Linux may soon be GPL3. If these separate pieces of inter-operable software are distributed together under two incompatible licenses (GPLv2 is incompatible with GPLv3, http://www.gnu.org/licenses/license-list.html#GNUGPL), then that would prove that just because B relies on, and is distributed with GPL'd A, does not mean that B is required to have the same, or even a compatible, license as A.

The possibility that proprietary software may in some circumstances use GPL'd parts is explored a bit on GNU's site:
"in many cases you can distribute the GPL-covered software alongside your proprietary system. To do this validly, you must make sure that the free and non-free programs communicate at arms length, that they are not combined in a way that would make them effectively a single program."
(from http://www.gnu.org/licenses/gpl-faq.html#GPLInProprietarySystem)

So, the trick to keeping an original work closed, when part of its function depends upon other GPL'd work, is to maintain separation between the projects. How this is technically achieved is not completely clear.

In the case of PHP or other interpreted languages, the "include" and "require" statements that one might use to bring in some functionality from GPL source (like a template engine or WYSIWYG editor) do not necessarily involve the same level of integration as static or dynamically linked libraries, which GNU advocates have argued clearly results in all parts combining into a single GPL whole.

Whatever technical means used in keeping separation, if you want your source to stay closed, the functions performed by the GPL software should not be the core functions of your proprietary program. For example, if you have some proprietary data manipulation software and would like to add a graph to a report it generates, it might be fine to distribute a GPL graphing script alongside your program without it necessarily "infecting" your closed license and forcing it open. However, if you were producing some closed reporting software whose primary or significant function was generating graphs, then you could not include GPL graphing software to perform that function and expect to keep your source closed.

Another requirement that I've assumed this far in the discussion is that your closed project actually is an original work, and not a "derivative" one. This is because the obligations imposed by the GPL only arise when a work is derived from the GPL work. Derivative is a term defined relatively loosely by US copyright law but involves the same analysis as with books and movies. For example, if I were to write a novel describing the adventures of Harry Potter after his 27th birthday, my work would be derived from the fictional universe created and copyrighted by J.K. Rowling, and since her work is not GPL, my work would be infringing. However, if I were to write a novel about a fictional boy who reads the Potter series and spends the rest of his life trying to learn magic, I am much more likely to have created an original, non-derivative work, deserving of independent copyright.

Whether your software that uses a GPL component is derivative of that component, therefore, depends. Were portions of your work copied from GPL source? Is your version just a wrapper around a GPL core? Or is your work a distinct entity that happens to be able to interact with GPL software?

It is possible to write non-free software that talks back and forth with GPL software, without losing your proprietary license status. The boundary is just fuzzy how close the relationship with GPL software can be, beyond which you will infect your closed project. If it is too close, the risk is that your work will be deemed legally "derivative" of the GPL one, and forced open by the GPL. But if you make sure the two projects keep their distance, talk at arms' length, and keep your project's core purpose distinct from the GPL one, then you will have created a new, non-derivative, copyrighted work and can avoid any sudden GPL "infection."

Of course, there is still much uncertainty in this area. For example, some companies refuse to write drivers for Linux because they believe (whether because of FUD or otherwise) doing so would cause the GPL to spread through their intellectual property like a disease, opening source to competitors for free, and harming themselves financially. After this analysis, I don't think that would actually be the legal outcome, but there's not a black and white answer. Perhaps with more research, there would be.

References:

http://blog.lab49.com/archives/659
http://drupal.org/node/25768
http://www.linuxjournal.com/article/6366
http://www.linuxjournal.com/article/5935
http://www.redhat.com/magazine/007may05/features/compliance/
http://tech.amikelive.com/node-14/the-gpl-myth-opensource-is-free-of-charge/
http://www.techdirt.com/article.php?sid=20070921/145609
http://radar.oreilly.com/archives/2007/07/the_gpl_and_sof_1.html
http://www.gnu.org/licenses/gpl-faq.html
http://www.gnu.org/philosophy/pragmatic.html
http://en.wikipedia.org/wiki/Open_source_vs._closed_source

Thursday, August 30, 2007

Dog Boots don't exist for a 80lb. steel spring

Review of Bark'n Boots Grip Trex

These boots display great quality materials (Vibram sole, neoprene/cordura uppers), smooth stitching and great craftsmanship. They are far better than anything else available such as Walkaboot, Ultra-Paws, Neopaws, etc). This whole shoe looks as good as any made for human children. HOWEVER, they did not fit or stay on, at least for my dog, who is like a tightly wound spring and creates a lot of traction forces when he runs and darts about.

The general design of these boots is still the same as the old version (which you may see on the clearance rack at some shops) in that the shoes only come up to the wrist, and unlike a human wrist or ankle, the width of the dog's wrist is about the same as the paw's width. That means this boot's single wrist strap holds about as well as a handcuff would on a cigar. My dog lost the first boot in less than 5 minutes after we started walking a wooded trail and continued to lose more at regular intervals.

Another problem with the fit was that the boots would flip upside down on the dog's feet so that he was standing on the uppers instead of the sole. For the boots that were not totally lost, I had to keep resetting them on his feet every 5-10 minutes. What allows the boot to twist around like this is that the inside of the boot is shaped like a cone, which allows the boot to rotate around on the foot. If the upper material was cut flatter and had more of a wetsuit stretch, it might resist that spinning better.

Relating to the boots' ability to stay on is its "side-loading" design like a slipper, as opposed to "top loading" like a human boot. Because of this and the poor holding power of just the single Velcro strap, the boot just comes off the way a tube sock would if it were pulled down to just the ball of your foot and given a few shakes. If the boot were top loading, though, the L-shape angle of ankle to foot seam would help hold the boot on, plus there could be additional lacing up the ankle as there is on a human hi-top shoe.

Another problem with the low cut is that at the back of the boot, where ankle becomes paw, the shoe suffers from "plumber butt." That is, at the L bend between ankle and foot, the shoe material hangs open in the same way that the back of your own pants opens along your rear belt loop when you bend over. This gap at the back of the shoe allows debris such as weed seeds, foxtails, pebbles, sand and other itchy poky things to fall inside the shoe where they will irritate the dogs feet worse than having no shoes at all. This could be alleviated if the upper were cut to angle up the leg a bit more before being wrapped with a second strap. A second strap could prevent things from falling inside the shoe and assist in securing the boot on the foot.

I contacted Ruff Wear about the problems of this low-top design and they said they would be coming out with another model boot that will secure higher on the ankle in Spring or Summer 2008.

After observing the performance of the Velcro in the field, I found that with a dog that runs through all kinds of grass and brush, the "sticky" side of the Velcro quickly becomes clogged with debris, reducing its effectiveness at staying fastened. I think that the straps on these boots got clogged enough to weaken the connection enough that just brushing against things on the trail and the dog's flexing caused them to release, and make the boot just suddenly open and fall off. Perhaps old-fashioned laces would be better, or else quick-adjust buckles like are used on backpack straps. Velcro in this application seems to only be good for securing dangling slack strap.

I feel part of the reason the boots stayed on so poorly was because they were not fitting properly. If you intend to buy these boots, you need to have a look at the "alternate size chart" that is buried in the FAQ on Ruff Wear's website. It is slightly different than the more common size chart that you typically see displayed near these products and it may help you pick the right size.

Also buried in their FAQ is notice that most dogs' rear feet are smaller than their front feet. Because of this, their rear feet may take a smaller size than their front feet. So, if you want to get the right fit, try on some boots at an REI store first, or order boots 2 at a time from ruffwear.com (they sell individual boots now for $15 each). Otherwise, if you just buy a set of 4 boots all the same size, as they are sold at retail, you may end up with half being too big.

Because of all these problems, I had to be dealing with boots every 5-10 minutes on the hike, instead of enjoying ourselves on it. These boots are so expensive to replace, the fact that they do not stay on as designed currently is a big problem. That said, I still would much rather have a set of boots that worked than $60 or even $100.

These did not work for me. If your dog just prances gently along, or is old and moves slowly, they might work for you.