Hi folks,

I've been using CouchDB as a CDN for a few projects, as it allows you to store binary attachments, and then provides an HTTP interface directly to these documents. In the past, I've used Apache as a proxy to avoid exposing the database directly to the web. So, the user requests files at:

example.com/download/{docid}/{attachment}

which is then proxied to CouchDB's HTTP API:

127.0.0.1/database/{docid}/{attachment}

The benefit to this approach is that CouchDB handles the load of serving large files very well. The drawback is that Lasso is completely unaware of this traffic.

A project I'm working on right now requires robust traffic logging and much more stringent authentication when downloading certain files. Since both my logging and authentication layers are handled in Lasso, I thought I'd try and write a proxy to do what Apache used to do, but which would allow me to run arbitrary code to log the request and determine whether to serve the file.

The following snippet shows what I was able to achieve, but it needs some work:

http://pastebin.com/0SMXsUHT

The way this code words is: it performs a HEAD request against the CouchDB file endpoint so it can grab the headers that CouchDB would have sent with its response, and then parses and sets the Lasso headers to mirror them exactly. Once headers are set, then I use curl inside a loop to write the body data to a bytes object, and periodically send a chunk of data once the bytes object exceeds fcgi_bodyChunkSize. (Hat-tip to Jolle for outlining this requirement with his export-csv chunking technique)

This approach is working successfully, mostly. I haven't found any problems with small to medium-sized Word, PDF, PowerPoint, image or even WMV documents. They all load progressively, instead of waiting for Lasso to load the entire file into memory and then serve it. However, so far both FLV and MP4 files (approx 50MB) don't stream, and I can't figure out why.

My leading theory is that Apache's configuration is somehow interfering with the content Lasso is streaming, because it might treat FLV and MP4 files uniquely. In that case, I don't know how to change Apache's configuration. Alternatively, maybe the byte data in these files is breaking curl, because I haven't configured my curl request correctly. Beyond that, I'm stumped, and could use a few pairs of eyes on this code to see if I'm missing something obvious.

To try this proxying technique on any URL, copy the code provided in the pastebin link above, and set #COUCHDB_URL to any url that serves a media file. You may need to change line 23 to cherry-pick the headers you want Lasso to mimic, based on what your given server sends back.

Any and all comments / suggestions are welcome and sorely needed. Thanks!

Ari.



#############################################################

This message is sent to you because you are subscribed to
the mailing list Lasso Lasso@lists.lassosoft.com
Official list archives available at http://www.lassotalk.com
To unsubscribe, E-mail to: <Lasso-unsubscribe@lists.lassosoft.com>
Send administrative queries to <Lasso-request@lists.lassosoft.com>