Houdini to Maya using ply2vrmesh and V-Ray Proxy (.bgeo .bgeo.sc .bhclassic .abc .vrmesh)

[hotbox]

Hi,

I’m working on an animation including an 8,300 frame Houdini generated fluid simulation.

Until recently I've exported the meshes using Houdini to alembic, then referenced in Maya as V-Ray Proxy Objects.

This works fine for network rendering, but due to the 3 main alembic files being quite large, between 0.5 and 1.5 TB, it doesn't seem to work for distributed rendering. The render nodes don't return the buckets (they either do nothing or return black, I can't remember which and can’t check right now as the workstation is unavailable).

Is this because the nodes are trying to transfer the large alembic assets locally but can't due to the size? They don't have enough local storage to copy over TB of data.

I’ve tried turning off 'Transfer missing assets', hoping the assets would be read from the file server as all are referenced in the Maya scene file via UNC using the file servers IP, but this doesn't work.

Hopefully someone can advise?

To try and get around the issue I've looked to reference as vrmesh as vrmesh supports a file sequence.

I tried converting the Houdini .bgeo.sc compressed cache files to vrmesh but received errors, I thought perhaps as .bgeo.sc are already compressed so can't be re-compressed into the vrmesh format.

Although I received errors again when trying to convert the .bgeo files.

The errors were 'missing magic number' and another shorter error I've forgotten.

Turns out the more recent Houdini .bgeo cache files from Houdini are not supported by ply2vrmesh.

They've been around for a while, are they planned to be at some point?

So with a bit of fiddling in Houdini I exported to the old Houdini .bhclassic format, renamed to .bgeo, then converted to .vrmesh.

Frustrating having to export to an old file format, especially an uncompressed one. Per frame file sizes being:

.bgeo.sc 274MB
.abc 886MB
.bhclassic 959MB
.vrmesh 340MB (converted from .abc)
.vrmesh 337MB (converted from .bhclassic renamed to .bgeo)

To get around this I've tried exporting an alembic in Houdini from the Houdini .bgeo.sc cache, then converting the alembic to vrmesh, which works, but seems a bit crazy.

The final conversion takes a very long time due to ply2vrmesh being single threaded, is it planned or possible for it to be multithreaded? Or is the only option to run multiple instances of the process against different sections of the cache to be converted?

On taking the vrmesh files into Maya I've encountered a couple of issues...

The V-Ray Proxy Object Animation Preferences 'Start offset' option doesn't seem to work with vrmesh sequences? I can overcome by renumbering the sequence to the Maya frames but am I missing something or does the 'Start offset' just not work with vrmesh sequences?

Also, the vrmesh sequences containing the fluid meshes reference well - the viewport updating as expected. Another vrmesh sequence that contains a particle simulation (a Houdini generated white-water simulation exported to alembic then converted to vrmesh) doesn't, on test rendering the sequence is updating, just not in the viewport – unless I switch the 'Geometry to load' from GPU Mesh to Maya Mesh, or vice versa, then it updates in the viewport.

Any advice would be much appreciated!

[psanitra]

Hi, Im houdini/max user and deal with abc/houdini/vray a lot, my 5 cents:

Do you export alembic file per frame? i can`t imagine you have 1.5TB single frame mesh...

Can you possible split heavy object before export to multiple meshes and export into multiple abcs?

You can try to do "culling" based on camera field of view too if that`s. Or adaptive mesh resolution by distance from camera if that`s ocean..

The actual data you exporting, i assume it`s water mesh. Is that coming from particles/VDB originally? because if yes, you can use the vdb convert node adaptivly before export, which makes a huge different in polycount..

Another option is not to export mesh/geo to abc at all. But export VDBs, load them as vray volume grid. And let the vray render do the actuall meshing on render. That will make the file size much much smaller.

For sure there are plenty of options available just give it a try , good luck.

[hotbox]

Hi,

Thanks for the reply!

I thought that frame sequences weren't supported by the Alembic format, that said, I've read both that they are an "abuse" of the format and that they do work. Sounds like you use them then?

I'm exporting about 50 alembic files from Houdini, the fluid mesh is the big one (most of the others are quite small), the fluid mesh is split into 3 sections.

Yes the mesh is limited to the camera's field of view.

Yes the mesh is a from a fluid simulation and converted to VDB along the way, so I'll have a look at reducing the polycount thank you.

Oh I didn't realise exporting to VDB and loading as a VRayVolumeGrid was an option, I'll have a look at this. VRayVolumeGrid in Maya wasn't an option the last time I used it as light linking wasn't working but (as below) I believe that's now been fixed so I'll have a look...

https://forums.chaosgroup.com/forum/...dering-problem

Again many thanks for the reply!

[psanitra]

hi

I use both single file ABC and ABC sequences, both work. You just type the usual $F on the filename of alembic ROP and it will save each frame to it`s own file. I don`t see anything wrong with that, aka "abuse". Alembic can store both static and deforming gemetry, per frame, plus point cache if the topology is not changing. In your case the topology is changing every frame, so you pretty much saving 8300 unique meshes into single file. My preferable way is to always use file per frame/substep for heavy geo, as it makes stuff much lighter for networking. Of course for basic stuff, like camera, small animation, use single abc file as usual

[Paul Oblomov]

I was asking same last year No luck here. So, the only way is to export alembics in chunks, like 100 frames per chunk. You can also use vrfh, but it is quite buggy. vdb is also good idea, but mesh, created by this method is very very simple, so ya have to properly smooth the initial grid data inside H, delete all unneeded channels, check the boundaries (sometimes they are crazy), it is like using compression node, so it will crop inside voxels leaving only water surface.

I wish one day vrayproxy node will read bgeo.sc with blosc compression without pain with conversion. That will save so much time...

[kagemaru]

As Peter said, just output an alembic per frame and load with ### in the vrayproxy (i read them too in max, but i reckon in maya it's the same). It works very smoothly.
When dealing with network rendering, it's just the only viable option (cache per frame), independently from the file/sw used.
I too dream of a .bgeo reader with vrayproxy, even more so for full featured vray version for Houdini...

[ricoholmes]

Fascinating stuff with the sequential ABC rendering. I wasn't aware of it as an option. It sounds like it could well cure some of the issues we have here too.
We're not sourcing our Alembics from Houdini here, but either meshed NParticles or Phoenix. Does anyone know a way of exporting sequential ABC files from Maya itself ?

[hotbox]

hi

I use both single file ABC and ABC sequences, both work. You just type the usual $F on the filename of alembic ROP and it will save each frame to it`s own file. I don`t see anything wrong with that, aka "abuse". Alembic can store both static and deforming gemetry, per frame, plus point cache if the topology is not changing. In your case the topology is changing every frame, so you pretty much saving 8300 unique meshes into single file. My preferable way is to always use file per frame/substep for heavy geo, as it makes stuff much lighter for networking. Of course for basic stuff, like camera, small animation, use single abc file as usual

Yes network speed was one of the concerns but we ran a 100 frame test a couple of weeks ago, the first referencing a single Alembic, the second 100x vrmesh files, the single Alembic was faster, only by a few percent, but surprised me.

I guess it makes little difference as the same amount of data is being pulled per frame, either from part of a single Alembic or from a single vrmesh file.

Then we have SSD caching on our file server for regular access files, as this was a test of a small section (there is a limit of about 0.75TB on the SSD cache) the single Alembic would have been cached to SSD on the first request, the multiple vrmesh files would have been pulled from the SATA RAID each time.

[hotbox]

As Peter said, just output an alembic per frame and load with ### in the vrayproxy (i read them too in max, but i reckon in maya it's the same). It works very smoothly.
When dealing with network rendering, it's just the only viable option (cache per frame), independently from the file/sw used.
I too dream of a .bgeo reader with vrayproxy, even more so for full featured vray version for Houdini...

Why do you say per frame is the only option for network rendering?

Very much second V-Ray for Houdini, would be amazing!

I'll give Alembic files sequences a try then, could solve the distributed rendering issue which would be great.

Thank you all again for the replies!