Stream Wrappers and You

This past weekend, a team of core Drupal developers, including Peter Wolanin (pwolanin), Andrew Morton (drewish), Jon Stacey (jmstacey), Justin Randell (justinrandell), Jody Hamilton (Jody Lynn), Alex UA (Alex UA), and myself, met at the Zivtech headquarters in Philly to hammer out some hot media and file functionality. After much arm-wrestling, teeth-gnashing, and nail-biting, plus some uber-helpful, last-minute, remote spot checks courtesy of Darrel O'Pry (dopry), Dries Buytaert (dries) and Moshe Weitzman (moshe weitzman), we managed to get the initial patch for PHP Stream Wrappers committed to Drupal 7!

As exciting as that is (a year-and-a-half of work from over a dozen contributors), the hard part has just begun. Part of me wants to relax and go celebrate. The other part of me wants to hunker down and get to work, because we only have about six weeks until the Drupal 7 code freeze.

I imagine many readers have two big questions: Just what is a stream wrapper, anyway? And What's next (and how can I help)? Let's jump right in, then.

Just what is a stream wrapper, anyway?

PHP, the scripting language that powers Drupal, offers some basic file manipulation capabilities. These include such functions as file_copy, which copies a file from one location to another, and file_delete, which likewise does what it says.

PHP also allows a built-in way to override that functionality, by implementing a class known as a Stream Wrapper. Thus, you could create a Stream Wrapper for a CDN such as Amazon S3, and allow it to manage files as easily as though they were on your local server. Stream Wrappers are identified within Drupal by their scheme (similar to the well-known http://), so that the public and private file systems are known as public:// and private://. Thus, an S3 Stream Wrapper class might use the scheme of s3://.

With the system thus abstracted, it is relatively easy to extend the functionality in further and sometimes novel ways. For instance, a stream wrapper might be implemented for Flickr images (flickr://), YouTube videos (youtube://), or even articles from the New York Times (nytimes://).

What's next (and how can I help)?

Here's where you come in. I'm going to present a list of links where you can jump in and help boost development.

  1. File API Stream Wrapper Conversion. Unfortunately, PHP Stream Wrappers was just the first step. We plan next to replace the current file system with stream wrappers, allowing for a wide range of exciting new functionality.
  2. Create the Media Module. An initial boost for all of this came from earlier sprints with Arthur Foelsche (arthurf), Darrel O'Pry (dopry), and others, to create a media asset API, of which stream wrappers is just the inward-facing piece. The outward-facing piece is a robust API to implement a unified GUI for managing stream assets. Sneak peek at some exciting Media module mockups by Maarten Verbaarschot (mverbaar) of the D7UX team.
  3. Version 2 of Embedded Media Field. With stream wrappers now in core, and development of the Media module in full-swing, the thousands of sites using Embedded Media Field to allow editors to easily embed third party content will need to be completely reworked to work with the new system. We still need to create a roadmap, which may involve simply migrating all existing third-party content to FileField.
  4. Converting your favorite media module! If you maintain a media module you want to ensure easily interfaces with the new stream wrapper functionality, feel free to ask for help. If you use a popular media module, and are not sure of its future status, jump on its issue queue and ask the maintainer.

Drupal is an awesome community, and it is this ongoing spirit of collaboration that allows for awesome things like stream wrappers in core to happen! Thanks to all the developers responsible for making it possible. And future thanks to those of you inspired to jump in and help.

Brian Puccio wrote 4 years 37 weeks ago

How does this fit in with CCK's filefield? Will we use media.module instead of filefield or in conjunction with filefield? From my reading of the media.module project page, it seems to be the latter.

Media, as a namespace, implies to me videos, audio and maybe even photos. If I'm looking to manage my large collection of Word docs, PDFs or other large files I may not be really looking for a media

Aaron Winborn wrote 4 years 37 weeks ago

The plan is to put FileField in core for Drupal 7, replacing both the core Upload and the contributed FileField modules. Thus, pretty much all media and files in the future will depend upon FileField (which will in turn use Streams).

Yes, you hit on an argument in earlier discussions regarding the Media namespace. The goal of the project is to create a unified file browser for all streams, whether videos, audio, images, Word docs, PDFs, New York Time blog posts, node references, or what-have-you. Stream GUI, or Resources would be the best description for that, but would be fairly obscure to most people. Maybe File Browser would work, though then it's not as apparent that it also handles third party media.

Feel free to chime in if you have a better namespace idea! (The first proposed, by the way, was Rich Media GUI, or RiMeG.)

Brian Puccio wrote 4 years 37 weeks ago

(Sorry, I can't see how to reply to a certain comment here...)

Filefield in core is something I'm really looking forward to. Thanks for the clarification and for all the work you (and the rest of the sprinters) have done!

Anonymous wrote 4 years 37 weeks ago

Wow file handling will be amazing yes? I've turned down every request to use drupal for DMS since I started with 4.7. It just didn't cut it. With the possibility of fields on files, and now this ... kudos!

manuee wrote 4 years 37 weeks ago

Outstanding work guys!! let's make this happen!

Contact Us

About Aaron Winborn

Aaron Winborn was Advomatic's first full time hire in 2006, and is a very active leader in the Drupal community. His first book, Drupal Multimedia is now available from Packt Publishing.

AdvoTwitter