How to use more than 4 GB of RAM with 32-bit Windows

Until this year, I had only 3GB of RAM in my ThinkPad T61 laptop because the collective wisdom of the internet dictated that my 32-bit Windows XP OS couldn't take advantage of any physical memory beyond 3GB.

For the past 3 years I've also been running this machine without a Windows paging file (no "virtual memory"), which, by the way, is the only performant way to run Windows XP due to its overly-aggressive swapping behavior (but that's a different topic).  The only problem with having only 3 gigs of RAM and no swap file is that when your applications need more than 3 gigs, you're out of luck, and as applications have been getting heavier over the years and programmers have been getting lazy about optimizing their memory profile, 3 gigs wasn't cutting it for me any more. And since I don't like to assume that something can't be done without trying it first, I decided to see what happens if I just go ahead and buy more RAM.

A few months ago, I bought a new 4GB stick, which brought my system up to a total of 6GB of physical RAM.  As I replaced the screws and booted my laptop, I was excited to see the BIOS recognizing all 6 gigs (which wasn't even possible according to the ThinkPad T61 docs).  However, as I opened up Task Manager, my hopes were dashed.  Instead of the prior 2880 MB maximum commit charge I had with 3 gigs, it was now giving me 2900 MB with my 6 gigs.  Awesome - I just paid $70 for an extra 20MB of RAM!  I haven't seen this good a deal since 1997.  I sighed, and started daydreaming about upgrading to Windows 7 64-bit to take advantage of my new RAM - a purchase which would bring the cost of the extra 3 gigs up to $270 (awesome - at least we're up to 2006 in memory costs now).

TL;DR:

I am now using almost all of the physical memory on my 32-bit Windows XP machine.  I did it by installing the free Dataram RAMDisk software.  Dataram lets you create a virtual in-memory disk drive using physical memory beyond 4GB - memory which isn't otherwise usable by Win32!  Allocating a 1.5 GB paging file (a.k.a. "swap file") on the RAMdisk, I now have 4436MB at my disposal!  If I wanted to, I could actually upgrade my RAM to 8GB and be able to use as much as 7 physical gigs without using neither a hard disk drive for swap space nor having to upgrade to a 64 bit operating system.


Conclusion:

I've been using this setup of almost a week.  No crashes so far.  The performance is decent, but unfortunately still not as good as it was without any paging file at all.  Even though task switching with the RAM disk is orders of magnitude faster than using a hard disk drive, I still get the same annoying GUI refresh problems in non-native apps that you get when letting the horrendous Windows XP memory manager get its hands on a paging file.

Caution:

Hibernation will (obviously) not work when you keep the system paging file on a RAMdisk.  Windows will give you the dreaded BSOD when attempting to wake up from hibernation.  That's because hibernation mode relies on all in-memory data being saved to disk, and Windows isn't aware of the need to save the content of the RAMdisk.  So if you're going to be using this hack, it might be a good idea to disable hibernation in your system settings.  Sleep mode still works fine, so the lack of hibernation is only a problem for laptops with a drained battery.

Summary:


I described 3 hacks in this article:

1. If you have enough RAM for your apps, you can, and should, run Windows XP without a paging file by going to Control Panel > System > Advanced > Performance > Advanced > Virtual Memory > Change, and selecting "No paging file" for all drives.
2. You can install more than 4GB into a ThinkPad T61 despite what the manual says.
3. You can use software, such as Dataram RAMDisk, to take advantage of all your physical RAM on a 32-bit Windows system.

Follow me:

For more computer, social, and life hacks, you should follow me on Twitter.

Enabling SSL with Python on Cygwin (secures Google App Engine's appcfg.py tool)

I prefer to run the dev_appserver through Cygwin, which works great, but appcfg complains:

WARNING appengine_rpc.py:399 ssl module not found. Without the ssl module, the identity of the remote host cannot be verified, and connections may NOT be secure. To fix this, please install the ssl module from http://pypi.python.org/pypi/ssl

I tried to fix this last summer, but didn't get anywhere on Cygwin and gave up after a few hours.  Instead,  I was able to install the ssl module on my Windows installation of python (not Cygwin) and have been using the App Engine Launcher (Windows client) to deploy my apps.

Today I decided to give try again on Cygwin.  To my surprise, the solution turned out to be very simple, but I only realized this after I messed up my Cygwin installation and had to run setup.exe about fifty times with different settings before I fixed it.

So here is the process:

1. download the ssl-1.15.tar.gz from http://pypi.python.org/pypi/ssl and extract the archive
2. > cd ssl-1.15
3. > python setup.py build

If the build fails, complaining about some OpenSSL-related stuff missing, just run Cygwin's setup.exe and install the openssl-devel package (make sure you get the same version as your libopenssl package, which you will also need, by the way).  Now the build should succeed.

4. > python setup.py install

That should do the trick.  

Be careful: do not run the install command until the build command runs successfully and until you make sure that you have the same version of the libopenssl and openssl-devel packages in your Cygwin installation.

Automating Subscriptions with Paypal

We'll be discussing the following Paypal APIs here (these are the official Paypal names for those APIs):

1. HTML - the API for generating Paypal buttons (HTML forms with hidden inputs)
2. PDT  - "Payment Data Transfer" - the API for requesting transaction details after the customer has finished checkout using a Paypal button.  Your access to this API is controlled by your PDT token, which you get from your Paypal profile page.
3. API - this is the one they actually call "API" - it can be used to request payments from customers and perform various administrative actions (like issue refunds).  One frustrating limitation is that you can't directly use this API to check the status of subscriptions created using the HTML API (e.g. whether a subscription has been cancelled by the customer), but you can hack around this limitation due to the fact that you can use this API to obtain info for all transactions. You can, however use this API to obtain the status of "recurring payments" which are a newer feature, more powerful than "subscriptions" (two concepts are not compatible, it seems).  Your access to this API is controlled by your choice of (a) an RSA certificate or (b) username/password/token (they call this token "signature" for some reason).  Again, you obtain these credentials from your Paypal profile page.
4. IPN - "Instant Payment Notification" - an API which your server can use to receive callbacks from Paypal notifying you of events related to your account (e.g. customer makes a purchase, a subscription expires, etc.)  This seems to be only way to programmatically obtain information about subscriptions from Paypal, and unfortunately, if you missed the notification (e.g. because you hadn't implemented IPN on your server yet), you'll have to update your database manually.

You can set up subscriptions by programmatically generating HTML for Paypal subscribe buttons.  Here's what happens:

1. User navigates to your product page, so you render the Paypal subscribe button (it's an HTML form which you can customize; we'll refer to this as the "button form" from now on).
2. User clicks on the button, completes the order on paypal.com and is redirected back to the return URL you specified in your button form.  Paypal will attach a "tx" parameter to the query string of this URL: this is the transaction ID.
3. Your server can make a request to the PDT API and include this TX ID to obtain more info about the transaction (e.g. which item was purchased, for how much, the fees charged, etc.)

You should also set up an IPN listener on your server if you want to be notified about changes in the status of the subscription (e.g. cancellations), otherwise you'll have to go fishing for this info manually in your Paypal account's transaction history, unless you use the following hack I came up with:

Hack - Using the "API" API  to check the status of subscriptions:

1. Get the "subscr_id" field from PDT - that's the ID of the subscription (a.k.a. "recurring payment profile")
2. Invoke the TransactionSearch method of the API to get a list of all transactions by that customer's email address ("payer_email" field from PDT), for example: 
   curl https://api-3t.paypal.com/nvp -d "METHOD=TransactionSearch&EMAIL==<replace with your payer_email value>&STARTDATE=2005-12-22T08:51:28Z&VERSION=56.0&USER=<replace with your value>&PWD=<replace with your value>&SIGNATURE=<replace with your value>
3. Look for all returned transactions whose L_TRANSACTIONID* fields are equal to your "subscr_id" from step 1. You can piece together all the events regarding this subscription this way (e.g. "Created", "Canceled", etc.)

That pretty much brings you up to date with what I know to date :)

Paypal has lots of completely different APIs and it's easy to get lost in their rather disorganized documentation.  I think I now see the big picture, so I wrote this post for my own future reference and hopefully it helped you as well.

Setting cookies from Google App Engine

I was disappointed to realize that the google.appengine.ext.webapp.Response class, while in many ways similar to the WebOb Response class, doesn't support the latter's set_cookie method.

I thought about using python's Cookie module directly, but but the WebOb method looked much more convenient.  So I decided the smartest thing to do was to just get the source code for the cookie-related operations from WebOb and adapt it to Google App Engine.  Feel free to use this little hack in your projects.

Example Usage:

   class FooHandler(webapp.RequestHandler):  
    def get(self):  
     cookies = Cookies(self.response)  
     cookies.add_cookie('foo', 'bar', max_age=360, path='/',
                        domain='example.org', secure=True) 

Source Code (cookies.py):

 from Cookie import BaseCookie  
 from datetime import datetime, date, timedelta  
 import time  
   
 # Implements WebOb's set_cookie, unset_cookie, and delete_cookie methods which GAE's webapp framework is missing  
 # modified from original WebOB code found at http://python-webob.sourcearchive.com/documentation/0.9.8-1/response_8py-source.html  
   
 # author: Alex Epshteyn ( http://myprogblog.blogspot.com/ )  
   
 # Original license from WebOb framework, which also applies to this file:  
   
 # Copyright (c) 2007 Ian Bicking and Contributors  
 # Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:  
 # The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.  
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.  
   
 class Cookies(object):  
  """  
  Wraps the 'headers' field of a google.appengine.ext.webapp.Response instance with cookie methods probided for a WebOb response object.  
  Example usage:  
   class FooHandler(webapp.RequestHandler):  
    def get(self):  
     cookies = Cookies(self.response)  
     cookies.add_cookie('foo', 'bar')  
  """  
   
  def __init__(self, response):  
   """  
   Takes a GAE webapp Response object (i.e. the value of the 'response' field of a google.appengine.ext.webapp.RequestHandler instance)  
   """  
   self.headers = response.headers  
   
  def set_cookie(self, key, value='', max_age=None,  
          path='/', domain=None, secure=None, httponly=False,  
          version=None, comment=None, expires=None, overwrite=False):  
    """  
    Set (add) a cookie for the response  
    """  
    if isinstance(value, unicode) and self.charset is not None:  
      value = '"%s"' % value.encode(self.charset)  
    if overwrite:  
      self.unset_cookie(key, strict=False)  
    cookies = BaseCookie()  
    cookies[key] = value  
    if isinstance(max_age, timedelta):  
      max_age = max_age.seconds + max_age.days*24*60*60  
    if max_age is not None and expires is None:  
      expires = datetime.utcnow() + timedelta(seconds=max_age)  
    if isinstance(expires, timedelta):  
      expires = datetime.utcnow() + expires  
    if isinstance(expires, datetime):  
      expires = '"'+_serialize_cookie_date(expires)+'"'  
    for var_name, var_value in [  
      ('max-age', max_age),  
      ('path', path),  
      ('domain', domain),  
      ('secure', secure),  
      ('HttpOnly', httponly),  
      ('version', version),  
      ('comment', comment),  
      ('expires', expires),  
    ]:  
      if var_value is not None and var_value is not False:  
        cookies[key][var_name] = str(var_value)  
    self._add_cookie(cookies)  
   
  def _add_cookie(self, cookie):  
    if not isinstance(cookie, str):  
      cookie = cookie.output(header='').lstrip()  
      if cookie.endswith(';'):  
        # Python 2.4 adds a trailing ; to the end, strip it to be  
        # consistent with 2.5  
        cookie = cookie[:-1]  
    if cookie:  
      self.headers.add_header('Set-Cookie', cookie)  
   
   
  def delete_cookie(self, key, path='/', domain=None):  
    """  
    Delete a cookie from the client. Note that path and domain must match  
    how the cookie was originally set.  
   
    This sets the cookie to the empty string, and max_age=0 so  
    that it should expire immediately.  
    """  
    self.set_cookie(key, '', path=path, domain=domain,  
            max_age=0, expires=timedelta(days=-5))  
   
  def unset_cookie(self, key, strict=True):  
    """  
    Unset a cookie with the given name (remove it from the  
    response). If there are multiple cookies (e.g., two cookies  
    with the same name and different paths or domains), all such  
    cookies will be deleted.  
    """  
    existing = self.headers.get_all('Set-Cookie')  
    if not existing:  
      if not strict:  
        return  
      raise KeyError("No cookies at all have been set")  
    del self.headers['Set-Cookie']  
    found = False  
    for header in existing:  
      cookies = BaseCookie()  
      cookies.load(header)  
      if key in cookies:  
        found = True  
        del cookies[key]  
        self._add_cookie(cookies)  
      else:  
        # this branching is required because Cookie.Morsel.output()  
        # strips quotes from expires= parameter, so better use  
        # it as is, if it hasn't changed  
        self._add_cookie(header)  
    if strict and not found:  
      raise KeyError(  
        "No cookie has been set with the name %r" % key)  
   
 # this function is from WebOb's datetime_utils.py (http://python-webob.sourcearchive.com/documentation/0.9.8-1/datetime__utils_8py-source.html)  
 def _serialize_cookie_date(dt):  
   if dt is None:  
     return None  
   if isinstance(dt, unicode):  
     dt = dt.encode('ascii')  
   if isinstance(dt, timedelta):  
     dt = datetime.now() + dt  
   if isinstance(dt, (datetime, date)):  
     dt = dt.timetuple()  
   return time.strftime('%a, %d-%b-%Y %H:%M:%S GMT', dt)