Building a userspace filesystem in node.js
?
1 like?860 views
How to build a filesystem using the open-source FUSE library in node.js. Talk given at Node.js Interactive in Portland, 12/8/15.
![BUILDING A FILESYSTEM IN NODE.JS
var files = [
{
name: 'readme.md',
data: new Buffer('# test filen'),
}
];
A 'file' is just an item in this array: Buffer()-backed](https://image.slidesharecdn.com/claynodeinteractivetalk-151209170346-lva1-app6892/85/Building-a-userspace-filesystem-in-node-js-12-320.jpg)
![BUILDING A FILESYSTEM IN NODE.JS
readdir() {
// ...
for (var i = 0; i < files.length; i++) {
var buffer = files[i].data;
var hash = hasha(buffer).substring(0, 12);
files[i].name = `${hash}-${files[i].name}`;
files[i].hasFingerprint = true;
}
}
Let's rename the filename to be a hash!](https://image.slidesharecdn.com/claynodeinteractivetalk-151209170346-lva1-app6892/85/Building-a-userspace-filesystem-in-node-js-13-320.jpg)
![BUILDING A FILESYSTEM IN NODE.JS
releaseFile(filePath, fd) {
// Buffer is a WritableStream()
var buffer = this.files[fileIndex].data.getContents();
zlib.gzip(buffer, function(error, result) {
// create file if compression succeeded.
});
}
Like gzip?](https://image.slidesharecdn.com/claynodeinteractivetalk-151209170346-lva1-app6892/85/Building-a-userspace-filesystem-in-node-js-14-320.jpg)
Building a userspace filesystem in node.js
- 1. FrontendFS Creating a userspace filesystem in node.js Clay Smith, New Relic
- 2. A file system (or filesystem) is a way of storing all data on a data storage device. BUILDING A FILESYSTEM IN NODE.JS Simple English Wikipedia
- 3. BUILDING A FILESYSTEM IN NODE.JS Kernel Virtual ?File ?System EXT3 Hardware NFS userspace kernel a ?process https://en.wikipedia.org/wiki/Protection_ring ? (thanks ?@maykaynwd!)
- 4. BUILDING A FILESYSTEM IN NODE.JS Kernel Virtual ?File ?System EXT3 Hardware NFS FUSE userspace kernel a ?process fuse ?client
- 5. BUILDING A FILESYSTEM IN NODE.JS Kernel Virtual ?File ?System EXT3 Hardware NFS a ?process FUSE fuse ?clientuserspace kernel libfuse
- 6. BUILDING A FILESYSTEM IN NODE.JS fuse ?client.js good news: node is great at I/O
- 7. BUILDING A FILESYSTEM IN NODE.JS How to write a (virtual) filesystem in node.js CC ?A ?SA ?https://www.flickr.com/photos/ewedistrict/25323526
- 8. BUILDING A FILESYSTEM IN NODE.JS ? Use the fuse-bindings github project ? https://github.com/mafintosh/fuse-bindings ? Implement required filesystem operations ? Run your node.js code and mount on an existing directory. ? Success?!?!
- 9. BUILDING A FILESYSTEM IN NODE.JS read(filePath, fd, buf, len, pos, cb) { } write(filePath, fd, buf, len, pos, cb) { } create(filePath, mode, cb) { } open(filePath, flags, cb) { } readdir(filePath, cb) { } Good ?list ?for ?reference: ?https://www.cs.hmc.edu/~geoff/classes/hmc.cs135.201001/homework/fuse/fuse_doc.html#function-?\ purposes ? getattr(filePath, cb) { }
- 10. BUILDING A FILESYSTEM IN NODE.JS getattr(/) getattr(/cat.jpg) create(/cat.jpg, 33188) getattr(/cat.jpg) write(/cat.jpg, 0, len: 65536, buf.length: 65536, pos: 0 write(/cat.jpg, 0, len: 65536, buf.length: 65536, pos: 65536) // ... getattr(/cat.jpg) release(/cat.jpg, 0) Example: What does a file copy look like?
- 11. BUILDING A FILESYSTEM IN NODE.JS In-memory virtual filesystem backed by JS array (yikes)
- 12. BUILDING A FILESYSTEM IN NODE.JS var files = [ { name: 'readme.md', data: new Buffer('# test filen'), } ]; A 'file' is just an item in this array: Buffer()-backed
- 13. BUILDING A FILESYSTEM IN NODE.JS readdir() { // ... for (var i = 0; i < files.length; i++) { var buffer = files[i].data; var hash = hasha(buffer).substring(0, 12); files[i].name = `${hash}-${files[i].name}`; files[i].hasFingerprint = true; } } Let's rename the filename to be a hash!
- 14. BUILDING A FILESYSTEM IN NODE.JS releaseFile(filePath, fd) { // Buffer is a WritableStream() var buffer = this.files[fileIndex].data.getContents(); zlib.gzip(buffer, function(error, result) { // create file if compression succeeded. }); } Like gzip?
- 15. BUILDING A FILESYSTEM IN NODE.JS Performance profiling CC ?A ?SA ?www.flickr.com/photos/libertinus/9231731038 ?
- 16. Benchmarking disk or file system IO performance can be tricky at best. BUILDING A FILESYSTEM IN NODE.JS James Coyle
- 17. BUILDING A FILESYSTEM IN NODE.JS dd: "The Easy Route" ?time sh -c "dd if=/dev/zero of=./in- memory/tst.tmp bs=4k count=1000 && sync" ?409600 bytes written 1000 times... ?Result: 3 mb/s (node, FuSE fs)
- 18. BUILDING A FILESYSTEM IN NODE.JS tl;dr - use for fun dev tools, exporting APIs, system learnings
- 19. BUILDING A FILESYSTEM IN NODE.JS see also... ? https://github.com/Munter/fusile ? ^ the "real" frontendFS ? https://www.npmjs.com/package/mount-url ? github.com/mafintosh/torrent-mount
- 20. BUILDING A FILESYSTEM IN NODE.JS thanks! @smithclay require('afterparty') - tomorrow at New Relic PDX. https://bit.ly/nodeparty