Bullet Serialization - general

Physics APIs, Physics file formats, Maya, Max, XSI, Cinema 4D, Lightwave, Blender, thinkingParticles™ and other simulation tools, exporters and importers
Post Reply
Chris Simson
Posts: 6
Joined: Tue Nov 10, 2009 9:12 am

Bullet Serialization - general

Post by Chris Simson »

Hi,

I cannot find any generic Bullet serialization routines in the API (not conversion such as ColladaConverter, but pure serialization).

Therefore I am attempting to write a serialization system for Bullet. I am goind to associate a serialize() function with each Bullet class.

Currently I am on the serialize<btRigidBody>() function. The bool m_additionalDamping member of btRigidBody is declared as private and has no accessor functions. How is one supposed to write this out to disk? You might think therefore that its value is implied by the value of other members, such as m_additionalDampingFactor(btScalar, but I cannot work out how!

Please advise.

Chris Simson
User avatar
Erwin Coumans
Site Admin
Posts: 4221
Joined: Sun Jun 26, 2005 6:43 pm
Location: California, USA
Contact:

Re: Bullet Serialization - general

Post by Erwin Coumans »

It is a good idea to add (de)serialization to Bullet. I've been looking into this for a while, and I'm planning to provide some solution soon.

Let's add serialize and deserialize methods (similar to irrlicht for example), and provide a basic binary chunk format that is backward/forward compatible (based on the iff format but adding support for little/bit endian and 32bit/64bit similar to the Blender .blend format).
http://code.google.com/p/bullet/issues/detail?id=308

If you are running into issues on your side, please just make the members protected for now,
Thanks,
Erwin
Chris Simson
Posts: 6
Joined: Tue Nov 10, 2009 9:12 am

Re: Bullet Serialization - general

Post by Chris Simson »

20091116 Bullet serialization

Sorry for the length of this posting.

The attached project needs to reside in the ...\bullet-2.75\Demos directory in order to build correctly. Please let me know if you can’t get it to work.

Attached is the first-cut, debug only go at a serialization system for Bullet. I must apologise in advance for strange development comments (to myself!) I am afraid it does not yet offer compatibility with any file-type standards (eg IFF), although that could be worked-in at later stages, (eg by deriving different Archive classes - maybe an IFFArchive?). That said, the attached work is not trivial (it took a good while to complete!) It is based upon the open-source Boost library serialization code (www.boost.org)(although there are very few, if any, direct code copies - parts of has_new_operator and the HashedString code).

The attached system was developed on a Win XP Pro X86 platform, using MS Visual Studio 2005 (Service Pack 1) (sorry if this offends Linux purists!) No attempt has yet been made to make it cross-compatible. Bullet-2.75 is being used.

Please see what you think - constructive criticism and feedback is welcomed.

I am currently making the serialization functions as globals, so that they are not intrusive into the Bullet system proper. This has the added advantage that the serialization system can be loaded-in as an optional extra, if needed. Also, in its current early state of development, I feel it is best to keep it separate. This does however necessitate accessor functions to all of the private and protected member variables within the Bullet classes (or for the serialization system to be made a friend of each of the Bullet classes, which again is intrusive, ie it requires modification of the main Bullet classes code). For the most part, such accessor methods are provided, but there are just a few exceptions which I have come across, as identified below. Other suggestions for minor Bullet modifications are also suggested below:


____Suggested changes to Bullet (for the sake of serialization) ( 2 items ):____

1)Accessor functions required for private/protected members;
Marked with INACCESSIBLE() in the code.

2) ...\bullet-2.75\src\BulletCollision\CollisionDispatch\btCollisionObject.h :
Line 118 +
Currently:
enum CollisionObjectTypes
{
CO_COLLISION_OBJECT =1,
CO_RIGID_BODY,
///CO_GHOST_OBJECT keeps track of all objects overlapping its AABB and that pass its collision filter
///It is useful for collision sensors, explosion objects, character controller etc.
CO_GHOST_OBJECT,
CO_SOFT_BODY,
CO_HF_FLUID
};

Suggestion:
enum CollisionObjectTypes
{
CO_COLLISION_OBJECT =1,
CO_RIGID_BODY = 2,
///CO_GHOST_OBJECT keeps track of all objects overlapping its AABB and that pass its collision filter
///It is useful for collision sensors, explosion objects, character controller etc.
CO_GHOST_OBJECT = 3,
CO_SOFT_BODY = 4,
CO_HF_FLUID = 5
};

Reasoning:
In order to re-attach a base class pointer from an archive, type information needs to be stored within the archive. Where RTTI is available, no assumptions about the class need to be made. However, since btCollisionObject and btRigidBody are not RTTI compatible (why, I do not know), the getInteralType() function has been used to provide type information. If therefore the ordering of enum CollisionObjectTypes is changed, this will confuse any archive that has stored these type values; hence the need to make them explicit.

NOTES
#20091119 The code has been tested (superficially) for memory leaks, and is currently clear.
#You get an access violation error message if you attempt to close the application via the OpenGl window (as opposed to the console window). This is nothing to do with the serialization code (the other Bullet demos have the same behaviour).

SERIALIZATION TODO:
# versioning not yet implemented.
# error-condition processing (modify all the asserts etc)
# complete the ‘add code’ segments!
# expand to incorporate other collisionobject types other than just rigid bodies.
# expand to include serialization of the elements of the debugWorld (such as the btBroadphaseInterface, btCollisionDispatcher etc); I don’t currently know if this is either a good idea, or necessary.


Chris Simson
Attachments
BasicDemoPersist.zip
(73.87 KiB) Downloaded 589 times
User avatar
Erwin Coumans
Site Admin
Posts: 4221
Joined: Sun Jun 26, 2005 6:43 pm
Location: California, USA
Contact:

Re: Bullet Serialization - general

Post by Erwin Coumans »

Hi Chris,

Thanks for sharing your work. I had a look at it. Good point about the enums/flags, they should not be re-ordered.

Bullet serialization will be built-in (each class gets its own methods), and allows for custom archives (and versioning, 32/64bit, little/big endian support and forward/backwards compatibility.

We don't want to add BOOST, RTTI or STL dependencies, to keep the code simple, portable and compiling out-of-the-box in all kind of environments.
Thanks,
Erwin

By the way: closing the window is probably a Windows Glut issue. Maybe we should replace Glut console app by a Win32 windowed OpenGL app, the code is already available. For now, just press 'Q' to exit demos.
Chris Simson
Posts: 6
Joined: Tue Nov 10, 2009 9:12 am

Re: Bullet Serialization - general

Post by Chris Simson »

Hi Erwin,

Thanks for the prompt response.
But how far down the rabbit hole to go, asked Alice!

Ok, so a set of criteria:
1) No RTTI
2) No STL
3) No Boost
4) Member serialization functions, not globals
5) 32/64 bit support
6) little/big endian support
7) Allows for custom archives, so as allowing for different file formats, or to serialize to/from clipboard instead of a file etc.
8) Versioning support / support forward and backward compatibility

You are a Task Master, Sir!

Taking these in turn:

1) No RTTI
As you can see from the BasicDemoPersist program, it does not use rtti to serialize the btCollisionObject hierarchy. Runtime type information (whether provided by rtti or some other mechanism) is necessary as far as I can see in order to reconnect/reinstantiate via base class pointers. In order to gain (non-rtti) runtime type information, one must therefore make some kind of assumption about the class. At the moment the serialization code within BasicDemoPersist assumes that an ‘int getInternalType()’ member function is present. This seems to work for the btCollisionObject class hierarchy. However, I have just started to look at the code for the btTypedConstraint (derived from btTypedObject); a separate ‘int getObjectType()’ function is used.
Suggestion
Derive btCollisionObject (and all other class hierarchies, if there are any others that Ive not found yet) from btTypedObject, so that you have a single universal system for providing type information. A good idea also would be to provide a single ‘typeinfo enum’ for all of the classes, to avoid uniqueid duplication, eg:

Currently you have:
enum btTypedConstraintType
{
POINT2POINT_CONSTRAINT_TYPE=MAX_CONTACT_MANIFOLD_TYPE+1,
HINGE_CONSTRAINT_TYPE,
CONETWIST_CONSTRAINT_TYPE,
D6_CONSTRAINT_TYPE,
SLIDER_CONSTRAINT_TYPE,
CONTACT_CONSTRAINT_TYPE
};
and
enum CollisionObjectTypes
{
CO_COLLISION_OBJECT =1,
CO_RIGID_BODY,
///CO_GHOST_OBJECT keeps track of all objects overlapping its AABB and that pass its collision filter
///It is useful for collision sensors, explosion objects, character controller etc.
CO_GHOST_OBJECT,
CO_SOFT_BODY,
CO_HF_FLUID
};
and
enum BroadphaseNativeTypes
{
// polyhedral convex shapes
BOX_SHAPE_PROXYTYPE,
TRIANGLE_SHAPE_PROXYTYPE,
TETRAHEDRAL_SHAPE_PROXYTYPE,
CONVEX_TRIANGLEMESH_SHAPE_PROXYTYPE,
CONVEX_HULL_SHAPE_PROXYTYPE,
CONVEX_POINT_CLOUD_SHAPE_PROXYTYPE,
CUSTOM_POLYHEDRAL_SHAPE_TYPE,
//implicit convex shapes
IMPLICIT_CONVEX_SHAPES_START_HERE,

...


You could have:
Enum ObjectTypes
{
CO_COLLISION_OBJECT =1,
CO_RIGID_BODY =2,
CO_GHOST_OBJECT =3,
CO_SOFT_BODY =4,
CO_HF_FLUID =5,

POINT2POINT_CONSTRAINT_TYPE=6,
HINGE_CONSTRAINT_TYPE =7,
CONETWIST_CONSTRAINT_TYPE =8,
D6_CONSTRAINT_TYPE =9,
SLIDER_CONSTRAINT_TYPE =10,
CONTACT_CONSTRAINT_TYPE =11

BOX_SHAPE_PROXYTYPE =12,

... plus all other types listed here, with explicit, non-reorderable identifiers
};

By making type identifiers unique and explicit, this goes some way to satisfying criteria 8) above – forward and backward compatibility.

If rtti is not used, mechanism of providing a unique class identifier is needed. At the moment, as far as I can see, Bullet does not provide this. For example,
TETRAHEDRAL_SHAPE_PROXYTYPE
and CO_RIGID_BODY
both currently evaluate to 2.

What do you think?


2) No STL
As I understand it, this means not using any of the std:: classes or functionality. To be honest, I think you are missing out on a good thing. However, that said, I believe the serialization code only uses the std::map structure, and the for_each() function. These are easy enough to implement ‘bullet-bespoke’ (see “Algorithms”, R. Sedgewick (1988)).

3) No Boost
Again, missing out on a good thing in my opinion, since, like STL, the code is tested on many different platforms, and as far as I am aware is completely open-source.
That said however, the serialization code provided has no dependency on Boost; it is based upon the Boost serialization framework, but does not use it. Ie the problem is NULL and void (sorry for the pun!)

4) Member serialization functions, not globals
This is not really an issue. The global functions could be made members. It is also possible to provide either-or (member or global), and for the serialization code to deduce at compile time which is provided for that particular class, and use the appropriate one (see the Chooser<>). This has the advantage non-intrusion if necessary.

5) 32/64 bit support
My knowledge of this issue is somewhat scant Im afraid. Could you enlighten me, or suggest a good source (book or url) please. If it simply means that, for example the btScalar type will be declared as a float or a double depending upon which architecture is used, I don’t really see this as a big problem – the system will work with all primitive types.
If you mean that, depending upon the architecture (32 or 64 bit) different primitive types will have different sizeof(type)s, eg is sizeof(int) on a 32 bit machine the same as on a 64 bit?
Suggestion
Include in the archive whether it is a 64 or 32 bit archive. That way the correct sizes for the data types can be assumed. But how do you interrogate the system to find out whether the system is 64 or 32 bit? Is there some kind of universally agreed function? Or do you just do a sizeof(int), and if it == 4, => 32bit ; sizeof(int) == 8 => 64 bit ??

6) little/big endian support
Again, little knowledge of this Im afraid. Could a good source be pointed out please. At a superficial level I understand this to determine the bit ordering of the types, so an int stored in big-endian fashion will have its most-significant bit at the other end of the block of memory to a little-endian implementation. Is this correct?
Suggestion
Again, the ‘endian-ness’ of the archive could be stored within it. But again, like the ‘bit-ness’, how do you gather this information (platform-independently) from the system? Is there a magic function?

7) Allows for custom archives
As I see it this is a matter of deriving from the Archive class different implementation, and reading/writing the data accordingly.
Again, the type of the archive could be stored within it, and a check be done to make sure that the correct archive class type is being used to read it.

8) Versioning support / support forward and backward compatibility
Do you version at the whole archive level, or at the individual class level? Either way, this means writing out the version number to the archive (either once for the whole archive, or once for each class instance). This information is read in and provided as a variable to the serialization function. The serialization function then modifies its loading behaviour depending upon the version number.
Thinking about it, an overall archive version number could be stored within the archive. This could be used to alter the behaviour of the archive system itself according to its value (ie so you can version the archive system itself). Then, if necessary, each serialization function could implement its own versioning (by reading in its own, class-level version number). This would then cater for both classes that have different versions, but also have the efficiency of not making it a prerequisite, and hence not serializing a version number for each class instance if it is superfluous.

Problems with Glut
Indeed, re the window-closing issue: I have implemented a non-glut OpenGL window and do not get these issues; the problem seems to be with glut. The bigger problem with glut is regarding checking for memory leaks - because the glutmain function does not return, this proves to be an issue when debugging for them. Is there a way around this? By the way, I also heard that the licence for it is not strictly open-source – a guy called Mark Kilgard retains it (seehttp://www.opengl.org/resources/libraries/glut/).


I’m afraid there are quite a few questions and suggestions there. However, with this additional information I feel that the serialization system could be hammer into something shipshape.

Chris Simson
Chris Simson
Posts: 6
Joined: Tue Nov 10, 2009 9:12 am

Re: Bullet Serialization - general

Post by Chris Simson »

Hi,

Have just thought of something else - as well as a serialize function, for classes with no default constructor, like btRigidBody, it may be necessary to include something like:

template<class Arch>
inline void load_construct_data( Arch & ar, btRigidBody * t, const UINT file_version)
{
btScalar mass;
ar >> mass;
btMotionState *pms = NULL;
ar >> pms;
btVector3 inertia;
ar >> inertia;
btCollisionShape *pcs = NULL;
ar >> pcs;
btRigidBody::btRigidBodyConstructionInfo info(mass, pms, pcs, inertia);
::new(t)btRigidBody(info);
}

It is necessary that this is either global, or a static member since it is called in order to undertake the instantiation via a placement new operator.

The other option is just to make sure that all classes you want to serialize have default constructors, ie have no arguements.

What are your thoughts?

Chris
User avatar
Erwin Coumans
Site Admin
Posts: 4221
Joined: Sun Jun 26, 2005 6:43 pm
Location: California, USA
Contact:

Re: Bullet Serialization - general

Post by Erwin Coumans »

We started the binary serialization code, and there will be a Bullet/Demos/SerializationDemo in Bullet 2.76.

Thanks,
Erwin
Post Reply