�ݺ�ߣ smallfiles

Feb 24, 20170 likes268 views

The document proposes a solution to replace inode-based storage with a key-value store mapping objects directly to positions in large "volumes" or files to address scalability issues. It benchmarks significantly better performance for puts, gets, and concurrent operations compared to an XFS filesystem, using less RAM and avoiding compaction costs. Open tasks include replication, erasure coding, and testing on object servers.

Issue
● 10-20 Millions object per devices
– 50 millions inodes per devices
● 36 devices per server
● 64 GB of RAM
– 1 inode is 1KB in RAM
– Would need 1.75TB of RAM for caching all inodes
● 75��% cache miss on inodes
– Up to 50��% of IO to get inodes from device
– (replicator/reconstructor constantly scan device...)

Solution
● Get rid of inodes
● Haystack-like solution
– Objects in volumes (a.k.a. big files, 5GB or 10GB)
– K/V store to map object to (volume id, position)
● K/V is an gRPC service
● Backed by LevelDB (for now...)
● Need to avoid compaction issue
– fallocate(PUNCH_HOLE)
– Smart selection of volumes

Benefits
● 42 bytes per object in K/V
– Compared to 1KB for an XFS inode
– Fit in memory (20GB vs 1.75TB)
– Should easily go down to 30 bytes per object
● Listdir happens in K/V (so in memory)
● Space efficiency vs Block aligned (!)
● Flat namespace for objects
– No part/sfx/ohash
– Increasing part power is just a ring thing

Adding an object
1.Select a volume
2.Append objet data
1.Object header (magic string, ohash, size, …)
2.Object metadata
3.Object data
3.fdatasync() volume
4.Insert new entry in K/V (no transaction)
● <o><policy><ohash><filename> => <volume id><offset>
=> If crash, the volume act as a journal to replay

Removing an object
1.Select a volume
2.Insert a tombstone
3.fdatasync() volume
4.Insert tombstone in K/V
5.Run cleanup_ondisk_files()
1.Punch_hole the object
2.Remove the old entry from K/V

Volume selection
● Avoid holes in volumes to reduce compaction
– Try to group objects by partition
● => rebalance is compaction
– Put short life objects in dedicated volumes
● tombstone
● x-delete-at soon
– Dedicated volumes for handoff?

Benchmarks
● Atom C2750 2.40Ghz
● 16GB RAM
● HGST HUS726040ALA610 (4TB)
● Directly connecting to objet servers

Benchmarks
● Single threaded PUT (100 bytes objects)
– From 0 to 4 millions objects
● XFS��: 19.8/s
● Volumes��: 26.2/s
– From 4 millions to 8 millions objects
● XFS��: 17/s
● Volumes��: 39.2/s (b/c of not creating more volumes?)
● What we see (need numbers!)
– XFS��: memory is full��; Volumes��: memory is free
– Disks is more busy with XFS

Benchmarks
● Single threaded random GET
– XFS��: 39/s
– Volumes��: 93/s

Benchmarks
● Concurrent PUT, 20 threads for 10 minutes
avg 50% 95% 99% max
XFS 641ms 67ms 3.5s 4.7s 5.9s
Volumes 82ms 50ms 261ms 615ms 1.24s

Status
● Done
– HEAD/GET/PUT/DELETE/POST (replica)
● Todo
– REPLICATE/SSYNC
– Erasure Code
– XFS read compatibility
– Smarter volumes selection
– Func tests on object servers (is there any?)
– Doc

Lincoln Bryant from the University of Chicago gave a presentation on using Ceph for Large Hadron Collider data storage and analysis. Key points: - Ceph is used to store reconstruction data from the ATLAS experiment at the LHC and for analysis datasets. - Ceph provides scalable storage through an erasure coded CephFS that is mounted by XRootD servers for data access. - Ceph allows efficient data transfers from CERN to the University of Chicago site for regional analysis. - Future evaluations include using librados directly with XRootD and running Ceph and analysis jobs on the same cluster nodes using cgroups for resource control.

Declarative Infrastructure Tools Yulia Shcherbachova

��

Lofar python meetup jan9 2013Gijs Molenaar

��

This document discusses LOFAR (Low-Frequency Array), a radio telescope array in the Netherlands, and how Python is used for data processing tasks related to LOFAR such as transient detection. It describes how Python is used for distributed computation, image processing, source extraction from datacubes, database storage in MonetDB, and creating VOEvents to report astronomical observations. Python libraries like NumPy, SciPy, Django, and custom libraries are crucial to the data processing pipeline.

Cassandra 2.1 boot camp, CompactionJoshua McKenzie

��

The document discusses Cassandra compaction, which merges SSTables to remove duplicate or overwritten data and dropped deleted data. It describes different types of compactions like minor, major, and single SSTable compactions. Compaction strategies like SizeTieredCompactionStrategy and LeveledCompactionStrategy are covered, which determine what SSTables to compact. The code walkthrough explains classes involved in compaction like CompactionManager, CompactionTask, CompactionController, and how iterators are used to merge cells and partitions during compaction.

"Metrics: Where and How", Vsevolod PolyakovYulia Shcherbachova

��

LOFAR - finding transients in the radio spectrumGijs Molenaar

��

The document describes the transient detection pipeline used for LOFAR radio telescope data. The pipeline involves extracting metadata from image data, performing quality checks, source extraction from multiple frequency bands and polarization, associating sources across images, detecting transient signals, classifying signals, and storing results in a database. It uses distributed computation, source extraction algorithms, and a column-oriented database (MonetDB) to handle large volumes of data from the telescope in an automated way to search for transient astronomical events.

OpenNebulaConf2018 - OpenNebula and LXD Containers - Rubén S. Montero - OpenN...OpenNebula Project

��

In this talk we'll showcase the new support for LXD containers in OpenNebula. The talk will describe the basic functionality of the new drivers and will provide some hints on the integration internals. LXD support will be released in OpenNebula 5.8 and it will let you manage LXD containers in your cloud using the same interfaces as with VMs, leveraging all the OpenNebula ecosystem and functionality including: Marketplace, multi-tenancy or service composition with OpenNebula Flow.

CRIU: Time and Space Travel for Linux ContainersKirill Kolyshkin

��

Fedora Virtualization Day: Linux Containers & CRIUAndrey Vagin

��

This document discusses Linux containers and checkpoint/restore (C/R) functionality. It provides an overview of different types of virtualization including containers and virtual machines. It then focuses on C/R, describing how it allows saving and restoring process states. It outlines the history and key components of C/R, including how it works, interfaces it uses, and features supported in the Linux kernel to enable C/R. It also discusses testing and future plans for C/R.

“Show Me the Garbage!”, Understanding Garbage CollectionHaim Yadid

��

“Just leave the garbage outside and we will take care of it for you”. This is the panacea promised by garbage collection mechanisms built into most software stacks available today. So, we don’t need to think about it anymore, right? Wrong! When misused, garbage collectors can fail miserably. When this happens they slow down your application and lead to unacceptable pauses. In this talk we will go over different garbage collectors approaches in different software runtimes and what are the conditions which enable them to function well. Presented on Reversim summit 2019 https://summit2019.reversim.com/session/5c754052d0e22f001706cbd8

Mongodb meetupEytan Daniyalzade

��

This document discusses Chartbeat's use of MongoDB and Amazon EC2. Chartbeat stores real-time analytics data and historical data in MongoDB clusters running on EC2. They faced challenges with disappearing EC2 instances, poor I/O performance on EBS volumes, and unpredictable EC2 performance. To address these, Chartbeat uses replica sets for high availability, preallocates data to reduce fragmentation, and heavily monitors servers and MongoDB for issues. Automating processes and monitoring are important strategies for stable MongoDB on EC2.

nebulaconfPedro Dias

��

This document discusses Pedro Dias's experience with OpenNebula. It describes how he used OpenNebula at his hosting company PTisp to replace their previous IaaS solution. OpenNebula met their requirements of being open, avoiding vendor lock-in, and having a low maintenance stack. The document outlines some customizations made to OpenNebula, such as using LVM instead of cLVM and developing tools like ONE Janitor to help with storage and VM sanitization. It also mentions contributions to the OpenNebula project and developing a Node.js addon and multi-tenant web panel called Cloudash.

C* Summit 2013: Time-Series Metrics with Cassandra by Mike HeffnerDataStax Academy

��

This document discusses using Cassandra to store time-series metrics data. It describes how the schema was matched to storage by using a measurement column family with rows organized by metric ID and time. It also covers optimizing data expiration through techniques like TTL expiration, synchronized compactions, and leveraging immutable sstable modification times. Effective monitoring is emphasized as well, including dashboards to track the ring and using Cassandra log volumes to identify issues.

Распределенные системы хранения данных, особенности реализации DHT в проекте ...yaevents

��

В этом докладе будет описана система хранения данных Elliptics network, основной задачей которой является предоставление пользователям доступа к данным, расположенным на физически распределенных серверах с плоской адресной моделью в децентрализованном окружении. Распределенная система хранения данных, предоставляющая доступ к объекту по ключу (key/value storage), и в частности распределенная хэш-таблица (distributed hash table), является весьма эффективным решением с незначительным набором ограничений. Для подтверждения работоспособности данной идеи и функционала в докладе будет представлена практическая реализация распределенной хэш-таблицы с модульной системой хранения данных и различными системами доступа: от POSIX файловой системы до доступа по протоколу HTTP. Также мы обсудим ограничения, накладываемые технологией распределенной хэш таблицы, и сравним особенности высоконагруженного и высоконадежного доступа в ненадежной среде с классическими моделями, использующими централизованные системы. Опираясь на полученные практические результаты и гибкость реализованной системы, будут предложены способы решения поставленных задач и расширения функционала.

Be a Zen monk, the Python waySriram Murali

��

CRIU: time and space travel for Linux containers -- Kir KolyshkinOpenVZ

��

tokyotalkHiroshi Ono

��

This document discusses Tokyo Cabinet and Tokyo Tyrant, which are key-value databases and a remote service wrapper. Tokyo Cabinet supports hash tables, B-trees, fixed-length arrays and tables. Tokyo Tyrant adds network support and features like high concurrency, multiple protocols, hot backup and replication. The document demonstrates installing and using both with Python via the pytc and pytyrant bindings. Performance tests show Tokyo Cabinet hash tables and B-trees have similar speed while Tokyo Tyrant has slower network overhead compared to direct Tokyo Cabinet access.

.NET Memory PrimerMartin Kulov

��

Ceph Day NYC: Developing With LibradosCeph Community

��

Lab 01 03_16Hao Wu

��

The document discusses the status and plans for pipelines integrating T cell receptor (TCR) and B cell receptor (BCR) analysis. The current TCR pipeline involves preprocessing taking around 10 minutes, storage for 5 minutes, and analysis for half an hour, with a minor problem to fix. It plans to improve the TCR algorithm from mitcr to mixcr, which will support additional receptor types, and move the BCR pipeline from fasta to fastq format, which is almost complete. A framework is envisioned to handle both TCR and BCR pipelines where the downstream analysis would be similar but the upstream processes are very different currently.

Remora the another asdf.hyotang666

��

Remora is a lightweight alternative to ASDF for managing Lisp system definitions and dependencies. Benchmark tests show that Remora can load the Hunchentoot system faster than ASDF, taking 29.572 seconds versus 36.464 seconds for ASDF. However, Remora uses more memory, allocating 304,207,024 bytes compared to 313,210,864 bytes for ASDF. Running Remora's tasks in parallel can further reduce the load time for Hunchentoot to 12.257 seconds but increases memory allocation to 2,952,899,360 bytes.

Memory managementmitesh_sharma

��

This document discusses memory related issues in Android applications. It explains that each app runs in a separate process with limited memory based on the device. If an app demands more memory than the limit, it will crash. Memory leaks and handling large bitmaps can also cause issues. Tools like logcat, MAT, and adb commands can help debug memory problems by analyzing heap dumps and tracking allocations over time.

Garbage collectionMudit Gupta

��

The document summarizes how garbage collection works in Java. It describes the marking phase where referenced and unreferenced objects are identified. Unreferenced objects are then deleted in the normal deletion step. For better performance, referenced objects can also be compacted together. The document further explains generational garbage collection, where new objects are allocated to the young generation and aged objects are promoted to the old generation. Minor and major garbage collections handle each generation. Different garbage collectors, like serial, parallel, CMS and G1, are also summarized regarding their implementation and suitability for different applications.

An Introduction to PriamJason Brown

��

SqliteToRealmPluu love

��

This document discusses migrating an Android app's database from SQLite to Realm. It describes how the author modified their webtoon app source code on GitHub to replace SQLite components like SqlBrite with Realm and related libraries. Performance tests showed improvements like reduced insertion and search times when using Realm compared to the original SQLite implementation. Some challenges mentioned include Realm's larger APK size, potential out of memory errors with large transactions, and ensuring compatibility with different Realm versions. Overall, the document recommends considering Realm from the beginning of a new project rather than migrating later.

A survey on Heap Exploitation Alireza Karimi

��

This document provides an overview of the glibc heap implementation and exploitation techniques. It begins with introductions to memory allocation on the heap versus stack, as well as definitions of key concepts like chunks, bins, arenas, and allocation functions. The document then details how the glibc heap is organized and managed, including through use of lists, bins, and mmap calls. Various exploitation techniques are covered like double free, forged chunks, unsorted/large bin attacks. Real world examples of use-after-free bugs are presented. The document concludes with a discussion of ongoing work to improve heap security.

FOSDEM2015: Live migration for containers is around the cornerAndrey Vagin

��

CRIU is a tool for checkpoint and restore of processes in Linux. It began as a project in OpenVZ in 2011 to allow migration of virtual machines without downtime. CRIU works by dumping the memory and process state of running processes, transferring that data to another machine, and restoring the processes from that saved state. Recent improvements include iterative dumping to reduce freeze times during migration and integration with tools like P.Haul for live migration of containers between hosts. CRIU is widely used and has an active development community contributing to new kernel features and enhancements to CRIU's capabilities.

Mongo nyc nyt + mongodbDeep Kapadia

��

The document discusses lessons learned from using MongoDB at the New York Times over 6 months. It covers initial setup without backups or monitoring, improving to replication and monitoring, optimizing storage, backups, restores, querying, indexing and administration. Key lessons include using replication and backups, monitoring all aspects of MongoDB and storage, optimizing data and indexes for queries, and understanding data and access patterns.

Ippevent : openshift Introductionkanedafromparis

��

This document provides an introduction to Docker and Openshift including discussions around infrastructure, storage, monitoring, metrics, logs, backup, and security considerations. It describes the recommended infrastructure for a 3 node Openshift cluster including masters, etcd, and nodes. It also discusses strategies for storage, monitoring both internal pod status and external infrastructure metrics, collecting and managing logs, backups, and security features within Openshift like limiting resource usage and isolating projects.

BluestoreCeph Community

��

This document summarizes BlueStore, a new storage backend for Ceph that provides faster performance compared to the existing FileStore backend. BlueStore manages metadata and data separately, with metadata stored in a key-value database (RocksDB) and data written directly to block devices. This avoids issues with POSIX filesystem transactions and allows more efficient operations like enumeration, cloning, and splicing. BlueStore also implements its own "file system" called BlueFS to share block devices between RocksDB and data storage.

More Related Content

What's hot (20)

Fedora Virtualization Day: Linux Containers & CRIUAndrey Vagin

��

“Show Me the Garbage!”, Understanding Garbage CollectionHaim Yadid

��

Mongodb meetupEytan Daniyalzade

��

nebulaconfPedro Dias

��

C* Summit 2013: Time-Series Metrics with Cassandra by Mike HeffnerDataStax Academy

��

Распределенные системы хранения данных, особенности реализации DHT в проекте ...yaevents

��

Be a Zen monk, the Python waySriram Murali

��

CRIU: time and space travel for Linux containers -- Kir KolyshkinOpenVZ

��

tokyotalkHiroshi Ono

��

.NET Memory PrimerMartin Kulov

��

Ceph Day NYC: Developing With LibradosCeph Community

��

Lab 01 03_16Hao Wu

��

Remora the another asdf.hyotang666

��

Memory managementmitesh_sharma

��

Garbage collectionMudit Gupta

��

An Introduction to PriamJason Brown

��

SqliteToRealmPluu love

��

A survey on Heap Exploitation Alireza Karimi

��

FOSDEM2015: Live migration for containers is around the cornerAndrey Vagin

��

Mongo nyc nyt + mongodbDeep Kapadia

��

Fedora Virtualization Day: Linux Containers & CRIUAndrey Vagin

��

“Show Me the Garbage!”, Understanding Garbage CollectionHaim Yadid

��

Mongodb meetupEytan Daniyalzade

��

nebulaconfPedro Dias

��

C* Summit 2013: Time-Series Metrics with Cassandra by Mike HeffnerDataStax Academy

��

Распределенные системы хранения данных, особенности реализации DHT в проекте ...yaevents

��

Be a Zen monk, the Python waySriram Murali

��

CRIU: time and space travel for Linux containers -- Kir KolyshkinOpenVZ

��

tokyotalkHiroshi Ono

��

.NET Memory PrimerMartin Kulov

��

Ceph Day NYC: Developing With LibradosCeph Community

��

Lab 01 03_16Hao Wu

��

Remora the another asdf.hyotang666

��

Memory managementmitesh_sharma

��

Garbage collectionMudit Gupta

��

An Introduction to PriamJason Brown

��

SqliteToRealmPluu love

��

A survey on Heap Exploitation Alireza Karimi

��

FOSDEM2015: Live migration for containers is around the cornerAndrey Vagin

��

Mongo nyc nyt + mongodbDeep Kapadia

��

Similar to �ݺ�ߣ smallfiles (20)

Ippevent : openshift Introductionkanedafromparis

��

BluestoreCeph Community

��

BluestorePatrick McGarry

��

This document summarizes BlueStore, a new storage backend for Ceph that provides faster performance compared to the existing FileStore backend. BlueStore manages metadata and data separately, with metadata stored in a key-value database (RocksDB) and data written directly to block devices. This avoids issues with POSIX filesystem transactions and enables more efficient features like checksumming, compression, and cloning. BlueStore addresses consistency and performance problems that arose with previous approaches like FileStore and NewStore.

Ceph scale testing with 10 Billion ObjectsKaran Singh

��

BlueStore: a new, faster storage backend for CephSage Weil

��

Traditionally Ceph has made use of local file systems like XFS or btrfs to store its data. However, the mismatch between the OSD's requirements and the POSIX interface provided by kernel file systems has a huge performance cost and requires a lot of complexity. BlueStore, an entirely new OSD storage backend, utilizes block devices directly, doubling performance for most workloads. This talk will cover the motivation a new backend, the design and implementation, the improved performance on HDDs, SSDs, and NVMe, and discuss some of the thornier issues we had to overcome when replacing tried and true kernel file systems with entirely new code running in userspace.

“Show Me the Garbage!”, Garbage Collection a Friend or a FoeHaim Yadid

��

“Just leave the garbage outside and we will take care of it for you”. This is the panacea promised by garbage collection mechanisms built into most software stacks available today. So, we don’t need to think about it anymore, right? Wrong! When misused, garbage collectors can fail miserably. When this happens they slow down your application and lead to unacceptable pauses. In this talk we will go over different garbage collectors approaches and understand under which conditions they function well.

Let's talk about Garbage CollectionHaim Yadid

��

This document discusses garbage collection in Java. It begins by explaining the motivation for garbage collection in Java, such as avoiding memory leaks and heap corruption. It then covers the goals of garbage collectors, including minimizing memory overhead, maximizing application throughput while keeping pause times low. Different types of garbage collectors are described, such as serial, parallel, CMS, and G1 collectors. Key concepts like generations, GC roots, and safe points are also summarized.

BlueStore: a new, faster storage backend for CephSage Weil

��

BlueStore is a new storage backend for Ceph that provides faster performance compared to the existing FileStore backend. BlueStore stores metadata in RocksDB and data directly on block devices, avoiding double writes and improving transaction performance. It supports multiple storage tiers by allowing different components like the RocksDB WAL, database and object data to be placed on SSDs, HDDs or NVRAM as appropriate.

Ceph Tech Talk: BluestoreCeph Community

��

BlueStore is a new storage backend for Ceph that stores data directly on block devices rather than using a file system. It uses RocksDB to store metadata like a key-value database and pluggable block allocation policies to improve performance. BlueStore aims to provide more natural transaction support without double writes by using a write-ahead log. It also supports multiple storage devices to optimize placement of metadata, data and write-ahead logs.

Ceph and RocksDBSage Weil

��

The Hive Think Tank: Ceph + RocksDB by Sage Weil, Red Hat.The Hive

��

Rocking the Database World with RocksDB Sage Weil, Ceph Principal Architect, Red Hat Sage helped design Ceph as part of his graduate research at the University of California, Santa Cruz. Since then, he has continued to refine the system with the goal of providing a stable next generation distributed storage system for Linux. Specialties: Distributed system design, storage and file systems, management, software development.

BlueStore, A New Storage Backend for Ceph, One Year InSage Weil

��

BlueStore is a new storage backend for Ceph OSDs that consumes block devices directly, bypassing the local XFS file system that is currently used today. It's design is motivated by everything we've learned about OSD workloads and interface requirements over the last decade, and everything that has worked well and not so well when storing objects as files in local files systems like XFS, btrfs, or ext4. BlueStore has been under development for a bit more than a year now, and has reached a state where it is becoming usable in production. This talk will cover the BlueStore design, how it has evolved over the last year, and what challenges remain before it can become the new default storage backend.

An Efficient Backup and Replication of StorageTakashi Hoshino

��

This document describes WalB, a Linux kernel device driver that provides efficient backup and replication of storage using block-level write-ahead logging (WAL). It has negligible performance overhead and avoids issues like fragmentation. WalB works by wrapping a block device and writing redo logs to a separate log device. It then extracts diffs for backup/replication. The document discusses WalB's architecture, algorithm, performance evaluation and future work.

Hoard_2022AIM1001.pptx.pdfAshutoshKumar437302

��

The document summarizes the Hoard memory allocator designed for multithreaded applications. Hoard aims to provide fast, scalable memory allocation that avoids fragmentation and false sharing. It uses per-processor heaps to avoid contention and a global heap to balance memory usage. Superblocks of same-sized objects are allocated from the OS in pages and objects within are handed out with low synchronization. Experimental results show Hoard outperforms other allocators on speed, scalability, fragmentation and false sharing avoidance.

Computer Memory Hierarchy Computer ArchitectureHaris456

��

The document discusses computer memory hierarchy and cache organization. It begins by outlining the memory pyramid from fastest and smallest registers to largest but slowest hard disks. It then discusses cache organization including direct mapped, set associative and fully associative caches. The key points are: Caches aim to bridge the speed gap between fast processors and slow main memory. Caches exploit temporal and spatial locality to reduce average memory access time. Caches are organized into blocks and sets to store recently accessed data from main memory.

SUE 2018 - Migrating a 130TB Cluster from Elasticsearch 2 to 5 in 20 Hours Wi...Fred de Villamil

��

Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...Flink Forward

��

Let’s be honest: Running a distributed stateful stream processor that is able to handle terabytes of state and tens of gigabytes of data per second while being highly available and correct (in an exactly-once sense) does not work without any planning, configuration and monitoring. While the Flink developer community tries to make everything as simple as possible, it is still important to be aware of all the requirements and implications In this talk, we will provide some insights into the greatest operations mysteries of Flink from a high-level perspective: - Capacity and resource planning: Understand the theoretical limits. - Memory and CPU configuration: Distribute resources according to your needs. - Setting up High Availability: Planning for failures. - Checkpointing and State Backends: Ensure correctness and fast recovery For each of the listed topics, we will introduce the concepts of Flink and provide some best practices we have learned over the past years supporting Flink users in production.

TritonSort: A Balanced Large-Scale Sorting System (NSDI 2011)Alex Rasmussen

��

We present TritonSort, a highly efficient, scalable sorting system. It is designed to process large datasets, and has been evaluated against as much as 100 TB of input data spread across 832 disks in 52 nodes at a rate of 0.916 TB/min. When evaluated against the annual Indy GraySort sorting benchmark, TritonSort is 60% better in absolute performance and has over six times the per-node efficiency of the previous record holder. In this paper, we describe the hardware and software architecture necessary to operate TritonSort at this level of efficiency. Through careful management of system resources to ensure cross-resource balance, we are able to sort data at approximately 80% of the disks' aggregate sequential write speed. We believe the work holds a number of lessons for balanced system design and for scale-out architectures in general. While many interesting systems are able to scale linearly with additional servers, per-server performance can lag behind per-server capacity by more than an order of magnitude. Bridging the gap between high scalability and high performance would enable either significantly cheaper systems that are able to do the same work or provide the ability to address significantly larger problem sets with the same infrastructure.

Save Java memoryJavaDayUA

��

Java heap memory model has wasteful memory usage. References, object headers, internal collection structure, extra fields such as String.hashCode… This talk shows practical ways to reduce memory usage and fit more data into memory: primitive types, specialized java collections, bit packing, reducing number of pointers, replacing String with char[], semi-serialized objects… As bonus we get lower GC overhead by reducing number of references.

.NET Core, ASP.NET Core Course, Session 4Amin Mesbahi

��

Ippevent : openshift Introductionkanedafromparis

��

BluestoreCeph Community

��

BluestorePatrick McGarry

��

Ceph scale testing with 10 Billion ObjectsKaran Singh

��

BlueStore: a new, faster storage backend for CephSage Weil

��

“Show Me the Garbage!”, Garbage Collection a Friend or a FoeHaim Yadid

��

Let's talk about Garbage CollectionHaim Yadid

��

BlueStore: a new, faster storage backend for CephSage Weil

��

Ceph Tech Talk: BluestoreCeph Community

��

Ceph and RocksDBSage Weil

��

The Hive Think Tank: Ceph + RocksDB by Sage Weil, Red Hat.The Hive

��

BlueStore, A New Storage Backend for Ceph, One Year InSage Weil

��

An Efficient Backup and Replication of StorageTakashi Hoshino

��

Hoard_2022AIM1001.pptx.pdfAshutoshKumar437302

��

Computer Memory Hierarchy Computer ArchitectureHaris456

��

SUE 2018 - Migrating a 130TB Cluster from Elasticsearch 2 to 5 in 20 Hours Wi...Fred de Villamil

��

Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...Flink Forward

��

TritonSort: A Balanced Large-Scale Sorting System (NSDI 2011)Alex Rasmussen

��

Save Java memoryJavaDayUA

��

.NET Core, ASP.NET Core Course, Session 4Amin Mesbahi

��

Recently uploaded (20)

study of impact behaviour of dual material for energy absorptionAmitChauhan352669

��

chapter 6.Construction project scheduling techniques pdfJimma Technology Institute,Jimma University

��

Matrices for Bid No Bid for estimation.pptxAshishshesharajBhale

��

Smart Manufacturing with Unified NamespacePonraj RK

��

22PCOAM16 Unit 2 Session 13 Radial Basis Functions and Splines.pptxGuru Nanak Technical Institutions

��

Pertemuan 4 (Introduction of Sedimentary rock).pptxandreahasbullah

��

Fault_Detection_Using_ANNs_Presentation.pptxJeveshMagnani

��

METAL OXIDE FIELD EFFECT SEMICONDUCTOR-MOSFETpunithaece

��

Operations Management - Facility Location.pptxVirajPasare

��

Vernier Calipers an details discussion to measure theDr Mohd Aslam

��

Chapter 2.pdf Smith Chart and Impedance Matchingdathoang3243

��

Chemical-Process-Safety-Fundamentals-With-Applications-2nd-Ed..pptChemical Engineering Dept. NIT Rourkela-769008, Odisha, India

��

Process safety has seen a dramatic consolidation of concepts in the past few years. Chemical Process Safety, provides students and working engineers with the understanding necessary to apply these new concepts to safely design and operate any process. Long the definitive guide in the field, this edition fully reflects major recent advances in process safety technology and practice. Readers will find extensive new and updated coverage of relief sizing, hazards identification, risk assessment, and many other topics such as Safety culture Preventive and mitigative safeguards Toxicology, industrial hygiene, and source models Hazardous material dispersion Fires, explosions, and concepts for preventing them Reliefs and relief sizing Hazards identification and evaluation Risk analysis and assessment, including Layer of Protection Analysis (LOPA) Safety strategies, procedures, designs, case histories, and lessons learned Crowl and Louvar link key academic concepts to modern industrial practice, making this guide invaluable for all engineering students and for all working engineers

lecture 4MORTAR for construction works(2).pptSimeonWoyesa

��

Environmental impact assessments ppt.docBonsaHailu

��

Introduction-to-Stack-Instruction.pptxpptpptsuhas060606

��

J111111111111111111111111111111111111111query.pptxdkmishra2407

��

Angelika Dorosz - BIM School Expert course - Poland, Krakowbim.edu.pl

��

The Mumbai Metropolitan Region Development Authority (MMRDA) was established ...ivargarud

��

Mastering Algorithms: Time and Space Complexity in AlgorithmsAnusha10399

��

ELECTROMECHANICAL ENERGY CONVERSION PROCESS BY LAKSHYA PANDEY.pptxLakshya Pandey

��

study of impact behaviour of dual material for energy absorptionAmitChauhan352669

��

chapter 6.Construction project scheduling techniques pdfJimma Technology Institute,Jimma University

��

Matrices for Bid No Bid for estimation.pptxAshishshesharajBhale

��

Smart Manufacturing with Unified NamespacePonraj RK

��

22PCOAM16 Unit 2 Session 13 Radial Basis Functions and Splines.pptxGuru Nanak Technical Institutions

��

Pertemuan 4 (Introduction of Sedimentary rock).pptxandreahasbullah

��

Fault_Detection_Using_ANNs_Presentation.pptxJeveshMagnani

��

METAL OXIDE FIELD EFFECT SEMICONDUCTOR-MOSFETpunithaece

��

Operations Management - Facility Location.pptxVirajPasare

��

Vernier Calipers an details discussion to measure theDr Mohd Aslam

��

Chapter 2.pdf Smith Chart and Impedance Matchingdathoang3243

��

Chemical-Process-Safety-Fundamentals-With-Applications-2nd-Ed..pptChemical Engineering Dept. NIT Rourkela-769008, Odisha, India

��

lecture 4MORTAR for construction works(2).pptSimeonWoyesa

��

Environmental impact assessments ppt.docBonsaHailu

��

Introduction-to-Stack-Instruction.pptxpptpptsuhas060606

��

J111111111111111111111111111111111111111query.pptxdkmishra2407

��

Angelika Dorosz - BIM School Expert course - Poland, Krakowbim.edu.pl

��

The Mumbai Metropolitan Region Development Authority (MMRDA) was established ...ivargarud

��

Mastering Algorithms: Time and Space Complexity in AlgorithmsAnusha10399

��

ELECTROMECHANICAL ENERGY CONVERSION PROCESS BY LAKSHYA PANDEY.pptxLakshya Pandey

��

�ݺ�ߣ smallfiles

1. Issue ● 10-20 Millions object per devices – 50 millions inodes per devices ● 36 devices per server ● 64 GB of RAM – 1 inode is 1KB in RAM – Would need 1.75TB of RAM for caching all inodes ● 75��% cache miss on inodes – Up to 50��% of IO to get inodes from device – (replicator/reconstructor constantly scan device...)

2. Solution ● Get rid of inodes ● Haystack-like solution – Objects in volumes (a.k.a. big files, 5GB or 10GB) – K/V store to map object to (volume id, position) ● K/V is an gRPC service ● Backed by LevelDB (for now...) ● Need to avoid compaction issue – fallocate(PUNCH_HOLE) – Smart selection of volumes

3. Benefits ● 42 bytes per object in K/V – Compared to 1KB for an XFS inode – Fit in memory (20GB vs 1.75TB) – Should easily go down to 30 bytes per object ● Listdir happens in K/V (so in memory) ● Space efficiency vs Block aligned (!) ● Flat namespace for objects – No part/sfx/ohash – Increasing part power is just a ring thing

4. Adding an object 1.Select a volume 2.Append objet data 1.Object header (magic string, ohash, size, …) 2.Object metadata 3.Object data 3.fdatasync() volume 4.Insert new entry in K/V (no transaction) ● <o><policy><ohash><filename> => <volume id><offset> => If crash, the volume act as a journal to replay

5. Removing an object 1.Select a volume 2.Insert a tombstone 3.fdatasync() volume 4.Insert tombstone in K/V 5.Run cleanup_ondisk_files() 1.Punch_hole the object 2.Remove the old entry from K/V

6. Volume selection ● Avoid holes in volumes to reduce compaction – Try to group objects by partition ● => rebalance is compaction – Put short life objects in dedicated volumes ● tombstone ● x-delete-at soon – Dedicated volumes for handoff?

7. Benchmarks ● Atom C2750 2.40Ghz ● 16GB RAM ● HGST HUS726040ALA610 (4TB) ● Directly connecting to objet servers

8. Benchmarks ● Single threaded PUT (100 bytes objects) – From 0 to 4 millions objects ● XFS��: 19.8/s ● Volumes��: 26.2/s – From 4 millions to 8 millions objects ● XFS��: 17/s ● Volumes��: 39.2/s (b/c of not creating more volumes?) ● What we see (need numbers!) – XFS��: memory is full��; Volumes��: memory is free – Disks is more busy with XFS

9. Benchmarks ● Single threaded random GET – XFS��: 39/s – Volumes��: 93/s

10. Benchmarks ● Concurrent PUT, 20 threads for 10 minutes avg 50% 95% 99% max XFS 641ms 67ms 3.5s 4.7s 5.9s Volumes 82ms 50ms 261ms 615ms 1.24s

11. Status ● Done – HEAD/GET/PUT/DELETE/POST (replica) ● Todo – REPLICATE/SSYNC – Erasure Code – XFS read compatibility – Smarter volumes selection – Func tests on object servers (is there any?) – Doc

�ݺ�ߣ

�ݺ�ߣ smallfiles

Recommended

More Related Content

What's hot (20)

Similar to �ݺ�ߣ smallfiles (20)

Recently uploaded (20)

�ݺ�ߣ smallfiles