How to use more than 4 GB of RAM with 32-bit Windows

Until this year, I had only 3GB of RAM in my ThinkPad T61 laptop because the collective wisdom of the internet dictated that my 32-bit Windows XP OS couldn't take advantage of any physical memory beyond 3GB.

For the past 3 years I've also been running this machine without a Windows paging file (no "virtual memory"), which, by the way, is the only performant way to run Windows XP due to its overly-aggressive swapping behavior (but that's a different topic).  The only problem with having only 3 gigs of RAM and no swap file is that when your applications need more than 3 gigs, you're out of luck, and as applications have been getting heavier over the years and programmers have been getting lazy about optimizing their memory profile, 3 gigs wasn't cutting it for me any more. And since I don't like to assume that something can't be done without trying it first, I decided to see what happens if I just go ahead and buy more RAM.

A few months ago, I bought a new 4GB stick, which brought my system up to a total of 6GB of physical RAM.  As I replaced the screws and booted my laptop, I was excited to see the BIOS recognizing all 6 gigs (which wasn't even possible according to the ThinkPad T61 docs).  However, as I opened up Task Manager, my hopes were dashed.  Instead of the prior 2880 MB maximum commit charge I had with 3 gigs, it was now giving me 2900 MB with my 6 gigs.  Awesome - I just paid $70 for an extra 20MB of RAM!  I haven't seen this good a deal since 1997.  I sighed, and started daydreaming about upgrading to Windows 7 64-bit to take advantage of my new RAM - a purchase which would bring the cost of the extra 3 gigs up to $270 (awesome - at least we're up to 2006 in memory costs now).

TL;DR:

I am now using almost all of the physical memory on my 32-bit Windows XP machine.  I did it by installing the free Dataram RAMDisk software.  Dataram lets you create a virtual in-memory disk drive using physical memory beyond 4GB - memory which isn't otherwise usable by Win32!  Allocating a 1.5 GB paging file (a.k.a. "swap file") on the RAMdisk, I now have 4436MB at my disposal!  If I wanted to, I could actually upgrade my RAM to 8GB and be able to use as much as 7 physical gigs without using neither a hard disk drive for swap space nor having to upgrade to a 64 bit operating system.


Conclusion:

I've been using this setup of almost a week.  No crashes so far.  The performance is decent, but unfortunately still not as good as it was without any paging file at all.  Even though task switching with the RAM disk is orders of magnitude faster than using a hard disk drive, I still get the same annoying GUI refresh problems in non-native apps that you get when letting the horrendous Windows XP memory manager get its hands on a paging file.

Caution:

Hibernation will (obviously) not work when you keep the system paging file on a RAMdisk.  Windows will give you the dreaded BSOD when attempting to wake up from hibernation.  That's because hibernation mode relies on all in-memory data being saved to disk, and Windows isn't aware of the need to save the content of the RAMdisk.  So if you're going to be using this hack, it might be a good idea to disable hibernation in your system settings.  Sleep mode still works fine, so the lack of hibernation is only a problem for laptops with a drained battery.

Summary:


I described 3 hacks in this article:

1. If you have enough RAM for your apps, you can, and should, run Windows XP without a paging file by going to Control Panel > System > Advanced > Performance > Advanced > Virtual Memory > Change, and selecting "No paging file" for all drives.
2. You can install more than 4GB into a ThinkPad T61 despite what the manual says.
3. You can use software, such as Dataram RAMDisk, to take advantage of all your physical RAM on a 32-bit Windows system.

Follow me:

For more computer, social, and life hacks, you should follow me on Twitter.

Enabling SSL with Python on Cygwin (secures Google App Engine's appcfg.py tool)

I prefer to run the dev_appserver through Cygwin, which works great, but appcfg complains:

WARNING appengine_rpc.py:399 ssl module not found. Without the ssl module, the identity of the remote host cannot be verified, and connections may NOT be secure. To fix this, please install the ssl module from http://pypi.python.org/pypi/ssl

I tried to fix this last summer, but didn't get anywhere on Cygwin and gave up after a few hours.  Instead,  I was able to install the ssl module on my Windows installation of python (not Cygwin) and have been using the App Engine Launcher (Windows client) to deploy my apps.

Today I decided to give try again on Cygwin.  To my surprise, the solution turned out to be very simple, but I only realized this after I messed up my Cygwin installation and had to run setup.exe about fifty times with different settings before I fixed it.

So here is the process:

1. download the ssl-1.15.tar.gz from http://pypi.python.org/pypi/ssl and extract the archive
2. > cd ssl-1.15
3. > python setup.py build

If the build fails, complaining about some OpenSSL-related stuff missing, just run Cygwin's setup.exe and install the openssl-devel package (make sure you get the same version as your libopenssl package, which you will also need, by the way).  Now the build should succeed.

4. > python setup.py install

That should do the trick.  

Be careful: do not run the install command until the build command runs successfully and until you make sure that you have the same version of the libopenssl and openssl-devel packages in your Cygwin installation.

Automating Subscriptions with Paypal

We'll be discussing the following Paypal APIs here (these are the official Paypal names for those APIs):

1. HTML - the API for generating Paypal buttons (HTML forms with hidden inputs)
2. PDT  - "Payment Data Transfer" - the API for requesting transaction details after the customer has finished checkout using a Paypal button.  Your access to this API is controlled by your PDT token, which you get from your Paypal profile page.
3. API - this is the one they actually call "API" - it can be used to request payments from customers and perform various administrative actions (like issue refunds).  One frustrating limitation is that you can't directly use this API to check the status of subscriptions created using the HTML API (e.g. whether a subscription has been cancelled by the customer), but you can hack around this limitation due to the fact that you can use this API to obtain info for all transactions. You can, however use this API to obtain the status of "recurring payments" which are a newer feature, more powerful than "subscriptions" (two concepts are not compatible, it seems).  Your access to this API is controlled by your choice of (a) an RSA certificate or (b) username/password/token (they call this token "signature" for some reason).  Again, you obtain these credentials from your Paypal profile page.
4. IPN - "Instant Payment Notification" - an API which your server can use to receive callbacks from Paypal notifying you of events related to your account (e.g. customer makes a purchase, a subscription expires, etc.)  This seems to be only way to programmatically obtain information about subscriptions from Paypal, and unfortunately, if you missed the notification (e.g. because you hadn't implemented IPN on your server yet), you'll have to update your database manually.

You can set up subscriptions by programmatically generating HTML for Paypal subscribe buttons.  Here's what happens:

1. User navigates to your product page, so you render the Paypal subscribe button (it's an HTML form which you can customize; we'll refer to this as the "button form" from now on).
2. User clicks on the button, completes the order on paypal.com and is redirected back to the return URL you specified in your button form.  Paypal will attach a "tx" parameter to the query string of this URL: this is the transaction ID.
3. Your server can make a request to the PDT API and include this TX ID to obtain more info about the transaction (e.g. which item was purchased, for how much, the fees charged, etc.)

You should also set up an IPN listener on your server if you want to be notified about changes in the status of the subscription (e.g. cancellations), otherwise you'll have to go fishing for this info manually in your Paypal account's transaction history, unless you use the following hack I came up with:

Hack - Using the "API" API  to check the status of subscriptions:

1. Get the "subscr_id" field from PDT - that's the ID of the subscription (a.k.a. "recurring payment profile")
2. Invoke the TransactionSearch method of the API to get a list of all transactions by that customer's email address ("payer_email" field from PDT), for example: 
   curl https://api-3t.paypal.com/nvp -d "METHOD=TransactionSearch&EMAIL==<replace with your payer_email value>&STARTDATE=2005-12-22T08:51:28Z&VERSION=56.0&USER=<replace with your value>&PWD=<replace with your value>&SIGNATURE=<replace with your value>
3. Look for all returned transactions whose L_TRANSACTIONID* fields are equal to your "subscr_id" from step 1. You can piece together all the events regarding this subscription this way (e.g. "Created", "Canceled", etc.)

That pretty much brings you up to date with what I know to date :)

Paypal has lots of completely different APIs and it's easy to get lost in their rather disorganized documentation.  I think I now see the big picture, so I wrote this post for my own future reference and hopefully it helped you as well.