Tag Archives: GridFS

Managing files with NodeJS and MongoDB GridFS


via Managing files with Node.js and MongoDB GridFS.

The technology stack was pretty simple, Nginx serving us on the front of the battle field and passing some of the requests to the node.js web server. For data storage we went with MongoDB, we hopped on the NoSQL wagon and just rolled with it. And as in many other projects we needed some sort of file management. Did some research and had to give a try to gridfs. So we used gridfs to manage and serve images (and in the future also other files too).

GridFS is a specification for storing and retrieving files that exceed the BSON-document size limit of 16MB. Instead of storing a file in a single document, GridFS divides a file into parts, or chunks, and stores each of those chunks as a separate document.

Probably now you ask yourself, can I store files smaller than 16MB? the answer is YES. You can store any size of files, all the files are going to be chopped up in chunks. Should I use gridfs if my files are under 16MB? It depends on your use case, I would suggest you do to some tests and research before using it, maybe it’s enough to use the simple BSON model.

So for using gridfs the advantages are numerous, for example you can take advantage of load balancing and data replication features over multiple machines for storing files. Also your files are splitted in small chunks. So for example getting a portion of your video file should be fairly easy to do.

GridFS by default uses two collections: fs.files and fs.chunks to store the file’s metadata and the chunks. Each entry in the chunks collection represents a small piece from a file. A document from the chunks collection contains the following fields:

Part 2: BUILDING MONGODB APPLICATIONS WITH BINARY FILES USING GRIDFS


Click to Read

In part 1 we looked at some of the use cases that are appropriate for GridFS. In this post we take a closer look at how GridFS works and how you can use it in your apps.

GridFS stores large binary files by breaking the files into smaller files called “chunks” and saving them in MongoDB. It essentially saves you, the application developer, from having to write all the code to break large files up into chunks, saving all the individual chunks into MongoDB, and then, when retrieving the files, combining all the chunks back together. GridFS gives you all this functionality for free.

The way GridFS works is shown in Figure 1. GridFS uses two collections to save a file to a database: fs.files and fs.chunks. (The default prefix is “fs”, but you can rename it.) The fs.chunks collection contains the binary file broken up into 255k chunks. The fs.files collection contains the metadata for the document.