�ݺ�ߣ

RMod
1
Mariano Martinez Peck
marianopeck@gmail.com
http://marianopeck.wordpress.com/
Monday, August 22, 2011

THANKS a lot!!!
2
SummerTalk 2011
Student: Martin Dias
Mentor: Mariano Martínez Peck

Object references
3

Object graph
4

5
An objet graph serializer.

Serialize
6
Input: an object graph
Output: stream of bytes

Materialize(deserialize)
7
Output: an object graph
Input: stream of bytes

Once serialized...
8
Database File Memory Socket
Stream of bytes

Fuel’s main goals
Provide fast object serialization and materialization.
Be ﬂexible and easy to customize.
Have a good OO design, well tested and
benchmarked.
No need of special support from the VM.
Be a general purpose serializer.
Allow tools to be built on top of Fuel.
9

Key features
Fast serialization and materialization.
Class reshape support.
Serialization of any kind of object.
Cycles support.
Global objects references.
Buffered writing.
Support for some “hook methods”.
10

Key Characteristics
Pickle format.
Objects grouped in clusters.
Analysis phase before writing.
Stack over recursion.
Two phases for writing instances and
references.
Iterative graph recreation.
11

Pickle format
12
Invest more time in serialization so that objects can then be
materialized much faster.

Grouping objects in
clusters
13
“Similar” objects (they share writing/loading information) are
grouped together in clusters. The most common case, yet not the
only one, takes place when a class is a cluster for its instances.

14
Each jar has
a speciﬁc
type of
element

14
Each jar has
a speciﬁc
type of
element
Jars are in
order

14
Label:
- What’s inside?
- How much?Each jar has
a speciﬁc
type of
element
Jars are in
order

14
Label:
- What’s inside?
- How much?
Different sizes
and different
amounts of
elements
Each jar has
a speciﬁc
type of
element
Jars are in
order

14
Label:
- What’s inside?
- How much?
Different sizes
and different
amounts of
elements
Each jar has
a speciﬁc
type of
element
Jars are in
order
s/jar/cluster

14
Label:
- What’s inside?
- How much?
Different sizes
and different
amounts of
elements
Jars are in
order
s/jar/cluster
Each cluster
has a speciﬁc
type of object

14
Label:
- What’s inside?
- How much?
Different sizes
and different
amounts of
elements
s/jar/cluster
Each cluster
has a speciﬁc
type of object
Clusters are
in order

14
Different sizes
and different
amounts of
elements
s/jar/cluster
Each cluster
has a speciﬁc
type of object
Clusters are
in order
Label:
- Cluster ID
- Amount of
objects

14
s/jar/cluster
Each cluster
has a speciﬁc
type of object
Clusters are
in order
Label:
- Cluster ID
- Amount of
objects
Different sizes
and different
amounts of
objects

Pickle format basic
15
Stream

16
Why the pickle format is so fast in materialization?
Standard serializers Fuel pickle format

16
Why the pickle format is so fast in materialization?
Standard serializers Fuel pickle format
Recursive
materialization Iterative
materialization

Pickle advantages
Batch/Bulk/Iterative materialization.
Efﬁcient since types are stored and fetch only
once.
Fast because at materialization we know the
size of everything.
The generated stream is smaller.
More next....
17

There is no silver
bullet...
18
Fast serialization
(without pickle)
Slow serialization
(with pickle)

Fuel requires
Traversing the object graph.
Mapping each object to a speciﬁc cluster.
19
This is done in a phase before
serialization called “Analysis”.

Analysis phase
20
cluster1 aPoint
cluster2 x y
...
...
... ...
key (a cluster) value (a set)
1) Traverse
(#fuelAccept: aVisitor)
2) Fill dictionary
(#fuelSerializer)

Serialization
21
key (a cluster) value (a set)
serialize: anObject on: aStream
materializeFrom: aStream
ID
Cluster
A cluster deﬁnes how its objects
are serialized and materialized.
cluster1 aPoint
cluster2 x y
...
...
... ...

Stack over recursion
22
To traverse the object graph, Fuel uses a custom
stack implementation rather than a recursion.

Basic steps
Serialization
1. Analyze.
2. Serialize header.
3. Serialize instances.
4. Serialize references.
5. Serialize root.
Materialization
1. Materialize header.
2. Materialize instances.
3. Materialize references.
4. Materialize root.
23

Fuel for software(so far)
Moose export utility.
SandstoneDB persistence.
Pier kernel persistence.
Newspeak language.
Marea (my own research project!).
24

Future Work
Continue efforts on performance optimization.
Create a tool for loading class and trait
packages.
Support user-deﬁned Singletons.
Fast statistics/brief info extraction of a stored
graph.
Partial loading of a stored graph.
25

Future work 2
Enable to deploy serialization and
materialization behavior independently.
Support object replacement for serialization
and materialization.
Allow cycle detections to be disabled.
Partial loading.
26

Serialization of primitive objects
27
Memory based stream

27
Memory based stream
StOMP
SRP
Fuel

27
Memory based stream File based stream
StOMP
SRP
Fuel

27
Memory based stream File based stream
StOMP
SRP
Fuel
Fuel
SRP
StOMP

28
Materialization
of primitive objects

28
Fuel
ImageSegment
StOMP
Materialization
of primitive objects

Non primitive objects
29

29
StOMP
SRP
Fuel

29
Fuel
ImageSegment
StOMP
StOMP
SRP
Fuel

Links
Website: http://rmod.lille.inria.fr/web/pier/
software/Fuel
Issue tracker: http://code.google.com/p/fuel
Source repository: http://www.squeaksource.com/
Fuel
Continuous integration server: https://pharo-
ic.lille.inria.fr/hudson/job/Fuel/
30

Excellent performance without special support
from VM and good OO design.
Conclusion
for us
31

Fuel is a vehicle. It is infrastructure. You can
build cool stuff on top of it.
Conclusion
for YOU
32

Thanks!
RMod
Mariano Martinez Peck
marianopeck@gmail.com
http://marianopeck.wordpress.com/

34
Concrete example

Analysis phase
3620
cluster1 aRectangle
cluster2 anOrigin aCorner
cluster3 10
key (a cluster) value (a IdentitySet)
20 30 40
Analysis
phase

Serialization
instances
phase
cluster1 aRectangle
cluster3 10 20 30 40

Serialization
instances
phase
cluster1 aRectangle
cluster3 10 20 30 40
1 Rectangle 2 origi corne 1
instance variable names
instance
count
instance variable count
class name
cluster ID

Serialization
instances
phase
cluster1 aRectangle
cluster3 10 20 30 40
1 Point 2 x y 2
instance
count
class name
cluster ID

Serialization
instances
phase
cluster1 aRectangle
cluster3 10 20 30 40
1 Point 2 x y 2
54 10 20 30 40
instance
count
class name
cluster ID

cluster1 aRectangle
cluster3 10 20 30 40
aRectangle anOrigiaCorner20 40 10 30
instancesIndex
(IdentityDictionary)
1 5 7 3 2 4 6
54 10 20 30 40
Serialization
references
phase
instances
references

cluster1 aRectangle
cluster3 10 20 30 40
instancesIndex
1 5 7 3 2 4 6
54 10 20 30 40
2 3
Serialization
references
phase
instances
references

cluster1 aRectangle
cluster3 10 20 30 40
instancesIndex
1 5 7 3 2 4 6
54 10 20 30 40
2 3
4 5 6 7
Serialization
references
phase
instances
references

2 3
4 5 6 7
final stream
1 Point 2 x y 2
54 10 20 30 40
{instances
{references

2 3
4 5 6 7
final stream
1 Point 2 x y 2
54 10 20 30 40
{instances
{references
1{trailer
7 3{header clustersCount
objectsCounts
root

Materialization
instances
{
2 3
4 5 6 7
1 Rectangle 2 origin corner 1
1 Point 2 x y 2
5 4 1 2 3 4
{references
1
{trailer
7 3{header clustersCount
objectsCounts
root

�ݺ�ߣ

Clustered Serialization with Fuel

Recommended

More Related Content

Similar to Clustered Serialization with Fuel (8)

More from Mariano Martínez Peck (8)

Recently uploaded (20)

Clustered Serialization with Fuel