COLLADA to Google Protocol Buffer binary

Physics APIs, Physics file formats, Maya, Max, XSI, Cinema 4D, Lightwave, Blender, thinkingParticles™ and other simulation tools, exporters and importers
Post Reply
micahp
Posts: 4
Joined: Wed May 09, 2007 8:15 pm

COLLADA to Google Protocol Buffer binary

Post by micahp »

I've started a project to convert COLLADA dae files to a Google Protocol Buffer compatible binary format:

http://code.google.com/p/collada2protobuf/

The motive behind the project is to create a binary reflection of COLLADA xml documents for reasonably fast and memory efficient resource management in the game application space. Google Protocol Buffer IDL was chosen because it seems to have a simple and reasonably generated API from its IDL, is compatible across platforms, compilers, and languages, and seems to be efficient in both speed and memory -- though to be honest I have yet to use it in a full production environment. (NOTE: the protobuf IDL can be used standalone without requiring COLLADA)

The project is in extremely early stages, with the only support for physics part of the COLLADA spec, though with graphical elements coming soon. The current physics support is very limited -- for example there is currently no support for constraints or collision meshes.

-Micah
User avatar
Erwin Coumans
Site Admin
Posts: 4221
Joined: Sun Jun 26, 2005 6:43 pm
Location: California, USA
Contact:

Re: COLLADA to Google Protocol Buffer binary

Post by Erwin Coumans »

Interesting, I haven't heard of Google Protocol Buffers before. It looks like very basic parsing can be done within 100 lines of C code.

How complex is your parsing code?

I've been looking into another binary chunk based format, called IFF 85, originally developed by Electronic Arts back in 1985. There is, however, no automatic conversion between COLLADA and IFF yet. The writing and reading for IFF is checked into Bullet/Extras/iff.

Thanks,
Erwin
micahp
Posts: 4
Joined: Wed May 09, 2007 8:15 pm

Re: COLLADA to Google Protocol Buffer binary

Post by micahp »

Erwin Coumans wrote: How complex is your parsing code?
If you mean reading the protobuf and converting it into Bullet physics an example can be seen:

http://code.google.com/p/collada2protob ... ufTest.cpp

If you mean parsing the binary format -- what the protobuf IDL does is compile into C++ code which can then map the binary into a nice C++ data structure (and vice versa). There is some supporting library code as well which is pretty standard C++.
Erwin Coumans wrote:I've been looking into another binary chunk based format, called IFF 85, originally developed by Electronic Arts back in 1985. There is, however, no automatic conversion between COLLADA and IFF yet. The writing and reading for IFF is checked into Bullet/Extras/iff.
I've used IFF for years. The advantages of IFF are that it is fast and simple. Protocol Buffers are reasonably fast and they simplify the processing of the raw data (automagically serializing the binary to C++ classes and vice versa) but I wouldn't call it simple (though if you don't care how the hotdog is made, as they say, it is very simple -- just compile and go). Protocol Buffers also have some other advantages, such as not being brittle (requiring a complete recompile) when the data structures change in minor ways. Also, another big advantage is that the IDL can be compiled into most common programming languages (C++, Python, Ruby, C#, Java, AS3, JavaScript, etc, etc) so say your tools or exporters can be in one language (C#) and your runtime in another (C++).

Though after saying all that I have fallen out of love with using COLLADA for physics scene definitions. The COLLADA spec is just too open for interpretation and trying to write a pipeline that can handle every interpretation sucks. I found it easier to just write a custom plugin and exporter for whatever tool I decide to use for editing physics scenes. Currently I am finishing up a plugin for Sketchup for editing physics scenes and exporting directly to protobuf's -- I may release that as open source if anyone is interested.

Cheers,

-Micah
User avatar
Erwin Coumans
Site Admin
Posts: 4221
Joined: Sun Jun 26, 2005 6:43 pm
Location: California, USA
Contact:

Re: COLLADA to Google Protocol Buffer binary

Post by Erwin Coumans »

micahp wrote: If you mean reading the protobuf and converting it into Bullet physics an example can be seen:
http://code.google.com/p/collada2protob ... ufTest.cpp
Thanks, I'll have a look at it.
Though after saying all that I have fallen out of love with using COLLADA for physics scene definitions. The COLLADA spec is just too open for interpretation and trying to write a pipeline that can handle every interpretation sucks.
Unfortunately there is still no conformance test, but Khronos is working on that. The COLLADA spec should be unambiguous, but unfortunately most developers write their own parser that is not compliant to the spec. Have you tried to use the BulletColladaConverter and COLLADA DOM, and use its import/export as a reference?

What kind things do you think are open for interpretation? Can you help out, so we can clarify that in the spec?
Currently I am finishing up a plugin for Sketchup for editing physics scenes and exporting directly to protobuf's -- I may release that as open source if anyone is interested.
That sounds interesting. Yes, that would be useful. Does it work with the free version of Sketchup? Do you include a Bullet conversion?
Thanks,
Erwin
micahp
Posts: 4
Joined: Wed May 09, 2007 8:15 pm

Re: COLLADA to Google Protocol Buffer binary

Post by micahp »

Erwin Coumans wrote: Unfortunately there is still no conformance test, but Khronos is working on that. The COLLADA spec should be unambiguous, but unfortunately most developers write their own parser that is not compliant to the spec. Have you tried to use the BulletColladaConverter and COLLADA DOM, and use its import/export as a reference?
I've used the BulletColladaConverter in the past as a reference for a COLLADA tool chain at a previous company and used it as a reference for the COLLADA to protobuf. Please don't take any offense but I found the code messy and very difficult to read, especially compared to the rest of the Bullet engine. Maybe it is because I dislike the COLLADA DOM and I found it impossible to simply refactor and use for my own purposes.
What kind things do you think are open for interpretation? Can you help out, so we can clarify that in the spec?
Totally off the top of my head and in no particular order (I didnt take notes -- just got frustrated):

* How one interprets the use of physics model verses a rigid body can be confusing. A physics model should be a collection of rigid bodies and constraints (greater then one rigid body). If there is only one rigid body there should be not be a physics model.
* The mass and mass frame shouldn't be able to be defined in a rigid body instance.
* How the whole mass thing gets defined is weird -- why not just define the mass and the 3x3 inertia tensor?
* How physics materials are used in rigid body definitions and rigid body instances is odd (for example I find it odd that a rigid body instance can override the rigid body definition physics material)
* How you have to build transforms is annoying and not obvious. (just give me the 4x4 matrix please)
* The constraint definition is a nightmare -- please define the standard joint types and leave the generic constraint to the people who care about those things.
* Having to find the graphics node to get the rigid body or model instance transform is a pain (A physics object shouldn't *have* to be bound to a graphics object to have a position in space).
* No support for physics triggers (ghosts)
* Support for 2D physics would be nice.
* Support for point type would be nice (for particle systems or possibly a verlet style integrator).
* Better conceptual separation of collision verses physics would be nice.

I'm sure I have a lot more to list and probably forgot some better criticisms -- but like I said this is off the top of my head. I'm a fairly experienced physics engineer (I've worked for both MathEngine and Havok) and I did a lot of scratching of head trying to figure out the COLLADA physics schema and then reconcile that with actual implementations. Your average game engine or tools engineer I'd imagine would have a very difficult time...
Currently I am finishing up a plugin for Sketchup for editing physics scenes and exporting directly to protobuf's -- I may release that as open source if anyone is interested.
That sounds interesting. Yes, that would be useful. Does it work with the free version of Sketchup? Do you include a Bullet conversion?
Thanks,
Erwin
I could possibly write a quick demo on how to use it with Bullet. I'll try to see about releasing it in the next few weeks.
User avatar
Erwin Coumans
Site Admin
Posts: 4221
Joined: Sun Jun 26, 2005 6:43 pm
Location: California, USA
Contact:

Re: COLLADA to Google Protocol Buffer binary

Post by Erwin Coumans »

micahp wrote: I've used the BulletColladaConverter in the past as a reference for a COLLADA tool chain at a previous company and used it as a reference for the COLLADA to protobuf. Please don't take any offense but I found the code messy and very difficult to read, especially compared to the rest of the Bullet engine. Maybe it is because I dislike the COLLADA DOM and I found it impossible to simply refactor and use for my own purposes.
Indeed, using the COLLADA DOM is a 'bag of hurt', in writing the BulletColladaConverter I hope to spare this hurt to most people: they should just use the load and save method.
What kind things do you think are open for interpretation? Can you help out, so we can clarify that in the spec?
Totally off the top of my head and in no particular order (I didnt take notes -- just got frustrated):

* How one interprets the use of physics model verses a rigid body can be confusing. A physics model should be a collection of rigid bodies and constraints (greater then one rigid body). If there is only one rigid body there should be not be a physics model.
* The mass and mass frame shouldn't be able to be defined in a rigid body instance.
* How the whole mass thing gets defined is weird -- why not just define the mass and the 3x3 inertia tensor?
* How physics materials are used in rigid body definitions and rigid body instances is odd (for example I find it odd that a rigid body instance can override the rigid body definition physics material)
* How you have to build transforms is annoying and not obvious. (just give me the 4x4 matrix please)
* The constraint definition is a nightmare -- please define the standard joint types and leave the generic constraint to the people who care about those things.
* Having to find the graphics node to get the rigid body or model instance transform is a pain (A physics object shouldn't *have* to be bound to a graphics object to have a position in space).
* No support for physics triggers (ghosts)
* Support for 2D physics would be nice.
* Support for point type would be nice (for particle systems or possibly a verlet style integrator).
* Better conceptual separation of collision verses physics would be nice.
I agree with most points, the specification should have been more simple (overly complicated design by committee)
I could possibly write a quick demo on how to use it with Bullet. I'll try to see about releasing it in the next few weeks.
Excellent, I'm looking forward to try it out. I haven't played with Sketchup yet. Does it also allow to execute the simulation inside the modeler?

Thanks a lot for the feedback,
Erwin
Post Reply