Considering Houdini with VRay - A few Q's

[Joelaff]

Considering adding Houdini for use with VRay (along side Max). Curious how the whole rendering process works with Houdini and what sort of license setup is required.

VRay licenses we have should just work, right? They are universal to all DCCs?

What about Houdini? What sort of license is required to render on a farm with Deadline? SideFX support wasn't as helpful / knowledgeable regarding use with VRay; so I figured I would ask people here.

I assume Houdini has to export .vrscene files or similar for rendering with VRay? Does that export require additional Houdini licenses on the farm? With Max the vrscene export does not require any Max licenses.

Can someone give me a rundown of the render process?

Is the scene export horribly slow like it is with large Max scenes and rendering with VRay Standalone? Does it produce giant files every time you render?

Are things like instanced geo and particles handled properly without having to write out every single instance (e.g. will 100000 of the same tree in a forest make about the same sized file as just one of those trees?)

What sort of production gotchas are you running into working with Houdini and VRay (and deadline) ? For example with Max and TyFlow (Max's fancy third party particle system) complex particles setups make for huge vrscene files for rendering and very long exports. Is this the case with Houdini/VRay as well? What about rendering with Mantra or Karma? Is it the same?

Does everything work smoothly with Deadline on AWS (cloud rendering?) If not, how are people dynamically adding cloud resources when needed while still leveraging their local farm?

Thanks for any insight into the overall process from an outsider looking at joining the club.

[Morbid Angel]

What about Houdini? What sort of license is required to render on a farm with Deadline? SideFX support wasn't as helpful / knowledgeable regarding use with VRay; so I figured I would ask people here.
I assume Houdini has to export .vrscene files or similar for rendering with VRay? Does that export require additional Houdini licenses on the farm? With Max the vrscene export does not require any Max licenses.

We recently switched to houdini/vray from maya/vray. Main reason - autodesk. I was a loyal maya user for last 20+ years as well as 3ds max. We went through this process and I can help a bit here.

For houdini side of things you need 3 types of licenses. The reason for the difference is the cost. Houdini FX is the most expensive, while core is about 1/2-1/3 the price and engine is even cheaper then that, so depending on your studio's situation you may chose the licenses to suit your need but here is the low down: 1 - houdini FX license which is their full blown license, that allows you to simulate particles and other fx. 2 - houdini core, which allows you to do everything except simulations. 3 - houdini engine license. This is required to be able to export on the farm from houdini to .vrscene.
The way we have this going at our studio is: I have 1-2 FX artists using houdini FX to create fx such as fire, smoke, destruction, particles etc, then I have 1-2 lighting artists using houdini core to light those scenes and make them into proper setups for rendering. After that, houdini fx, houdini core can also be used as export license (houdini engine) on the deadline side. Depending on the scale of your farm, and the size of your team, you may or may not need houdini engine licenses. For us we also have 2 houdini engine licenses and we rent more depending on the demand. With 6 export licenses we are able to chug through most of the stuff with no problem.

Is the scene export horribly slow like it is with large Max scenes and rendering with VRay Standalone? Does it produce giant files every time you render?
Are things like instanced geo and particles handled properly without having to write out every single instance (e.g. will 100000 of the same tree in a forest make about the same sized file as just one of those trees?)

This is something very tricky in houdini that is unlike max or maya. You can export anything to vrscene, but you have to be smart. For example if you use alembics in your caches the export is fast, the file sizes are small. Yes instances get preserved so no duplications. If you use bgeo caches, such as volumes, particles etc, vray would have to convert it and write it out. This is a painful step in many ways. For example we have to convert volumes to vdb so that the export part is fast. We have to use always alembics and not bgeo (geo caches) for the same reason. We are currently working on a particle sim with few hundred thousand particles and about 200 instances, and the file size is 30 Mb/frame. Usually exporting a 100 frame sequence takes just a few minutes if you have a fast network. However this is a step unlike max or maya where you never have to worry about that unless you work with standalone. (Such as we do to save costs).

What sort of production gotchas are you running into working with Houdini and VRay (and deadline) ? For example with Max and TyFlow (Max's fancy third party particle system) complex particles setups make for huge vrscene files for rendering and very long exports. Is this the case with Houdini/VRay as well? What about rendering with Mantra or Karma? Is it the same?

There are a lot to cover here, the main thing is how to handle export outputs for vray, the export name syntaxes and how to work with rops Its very very different from anything else I've experienced out there, its powerful beyond max and maya but you have to change the way you think for sure. If you had for example unlimited houdini licenses, such as houdini engine to cover your entire farm then you don't even need to export to vray scene you can just render directly since during rendering the scene will be exported automatically but that does add to render time. For example writing single 1Gb volume frame. The way to go here however would be to rely on solaris, however we have explored that in depth and came to realization that its not supported in deadline natively. That is there is no submitter for it. So you have to submit manually and its a huge pain in the ass. Also solaris is powerful but it is very difficult.

Does everything work smoothly with Deadline on AWS (cloud rendering?) If not, how are people dynamically adding cloud resources when needed while still leveraging their local farm?

We don't export or send jobs to aws, but we use local deadline successfully for last 6 month on multiple shows. I can't speak more to the custom render farm environments however. All I can say is that for any machines that will participate in export jobs you will have to do a full houdini and vray installation with license activations before you are able to export the scenes for vray.
The way this works for us in the end is that, we have a rop per render layer, each rop is being exported to deadline and with vrayDeadline submitter it automatically creates two jobs, one is the export job, the second is the dependent render job. Once the frames are exported to .vrscene deadline automatically releases the dependency and they start rendering. It's pretty seamless for us with few minor but annoying limitations.

[Joelaff]

I appreciate your response. I saw your mention of switching to Houdini in other threads; so thanks for chiming in here. Your experience is part of what is making me consider this more carefully.

It sounds very much like working with Standalone from max, which we do when we want to use AWS for rendering, or think we might need to (due to the very high (like 2-5x more $) Microsoft Tax on AWS VMs). The local farm is a good size, and is normally adequate, but when some creative director doesn't bother commenting until the day before we ship (amazing how frequently this happens, and it's mainly with larger agencies, who seem to be the pinnacle of inefficiency in practically every way) and all shots have to be re-rendered overnight that is when we tap into AWS. So we have been through the use alembic for everything route, and the joy of getting everything we can to external files to keep the vrscene exports fast and as light as possible!

So to export to vrscene on the farm requires a Houdini Engine license AND a VRay license?!?! Is that a full interactive VRay license just to export for rendering? That would be ridiculous. In Max this step does not require a VRay license of any kind (or even a Max license, thankfully). Or does exporting to vrscene just require a Houdini Engine license>

When you export to vrscene are you then using Standalone to do the actual rendering? I mean, when you submit to deadline is it generating a VRay Standalone job once the export has happened?

So if you could export to deadline from Solaris would it be your preferred approach to rendering from Houdini? I guess I still don't fully understand what Solaris is or how it interfaces with Houdini-- Is it a standalone app? I would think a submitter wouldn't be too hard to code with Deadline; so I might look at building one, but there are currently none available. When you have tested rendering from Solaris do you have to produce a vrscene from Solaris and then submit that to Deadline? (which does sound like a pain).

Can all of these steps be done on either Windows or Linux? We would love to move to 100% Linux someday. That is the dream, to be rid of as much licensing BS and adware crap from MS as possible.

Slightly OT, but can the Houdini Engine license do standalone simulations for Houdini on the farm? Or do you actually need a full multi-thousand dollar UI Houdini FX license just to do headless sims?

Thanks again. I appreciate your time (and anyone else who wants to chime in with their experiences).

[Morbid Angel]

So to export to vrscene on the farm requires a Houdini Engine license AND a VRay license?!?! Is that a full interactive VRay license just to export for rendering? That would be ridiculous. In Max this step does not require a VRay license of any kind (or even a Max license, thankfully). Or does exporting to vrscene just require a Houdini Engine license>

Yes unfortunately as far as I can tell it will use one Houdini Engine and one VRay Render license. However not the interface license, and if you have a large farm or a farm chances are you have those licenses available. If the vrscene export is optimized and file size is small the write time of it is really fast. I mean with regular rendering like in 3ds max, renderer still loads all that data, its just already cooked for it. If you use any standalone this is a common process.

When you export to vrscene are you then using Standalone to do the actual rendering? I mean, when you submit to deadline is it generating a VRay Standalone job once the export has happened?

Yes you are rendering with vray standalone by that point, only vray render license will be needed. Deadline generates 2 jobs when you submit using vrayDeadline submitter. One is a houdini export job, one is a vray standalone job that's dependent on the export job.

So if you could export to deadline from Solaris would it be your preferred approach to rendering from Houdini? I guess I still don't fully understand what Solaris is or how it interfaces with Houdini-- Is it a standalone app? I would think a submitter wouldn't be too hard to code with Deadline; so I might look at building one, but there are currently none available. When you have tested rendering from Solaris do you have to produce a vrscene from Solaris and then submit that to Deadline? (which does sound like a pain).

Main issue we have with houdini is that it does not have any kind of reference system, no xref, no maya like reference. It only has digital asset which is a way to package a scene into an asset and load it to another scene, like (reference scene in 3ds max), where you can't modify it in any way and only place it. You can however unpack the digital asset and modify it thereby baking it to that scene, its a bit more complicated then that though but lets say you have a light rig with lights that you want to have lighters reference into scenes and be able to move lights around, change their values etc, its pretty hard to do that. So the main issue becomes on how you share master assets and maintain their versions.

Solaris is suppose to solve this issue. But whoever wrote it thought not like the likes of autodesk coders. The logic is very different and so is the rendering process. Its very very complicated. In one video tutorial some one said: This is a beginner Solaris tutorial, but beginner Solaris tutorial requires a Houdini Veteran. You will be diving into a deep rabit hole with solaris with end result uncertain. It's useful for large scale productions who have tons of money and dev time to throw at it, for smaller scale stuff not sure. Solaris is not a plugin, but it is a 3rd party implemented into houdini interface. All it does is allows you to load usd, modify them, assign shaders and other operators to them, write them out etc. The beauty here is that you can have assets as usd loaded into shots then swap just the cache of the asset (animation for example) onto it and have it render correctly etc.

Solaris on deadline renders through husk which is a custom render plugin like vray.exe. So you have to pass the render command to it and there is no out of the box support for submitting to husk from houdini. On thinkbox forum some one wrote something, but lots of people can't get it to work etc etc its a huge mess.

Can all of these steps be done on either Windows or Linux? We would love to move to 100% Linux someday. That is the dream, to be rid of as much licensing BS and adware crap from MS as possible.

Yes it should work on linux the same way, though I have not tested it as we are only on windows. We do not render on remote farms but I know the pain of windows vms.

Slightly OT, but can the Houdini Engine license do standalone simulations for Houdini on the farm? Or do you actually need a full multi-thousand dollar UI Houdini FX license just to do headless sims?

Thanks again. I appreciate your time (and anyone else who wants to chime in with their experiences).

Yes Houdini Engine can do sims on the farm - that's what its meant for. So a user would use Houdini FX to create the scene then submit it to sim using engine and get it simmed on a farm machine.

[Joelaff]

Ok. A VRay render license to export is not a huge deal, though annoying, since we frequently let the older hardware do the io tasks like exports, and leave the VRay licenses for the machines fast enough to render 3d.

Interesting about Solaris. We are mainly commercials, which have very little r&d time available, sadly, because I love doing things “right,” but that is rarely economical viable with commercial schedules.

Good on the engine licenses also for sims; at least they will be useful for something else.

Thanks again for all the insights. I guess I’ll have to hit them up for some eval licenses. I hate that floating licenses are so much more expensive, as this really hurts medium sized studios who need them, while subsidizing the folks that don’t. Large studios then get large volume pricing, with the medium folks being the only ones paying the high prices. But this seems to be everything these days.

[Morbid Angel]

It is like that to a degree for the floating licenses. In a smaller studio environment you can say ok we know this person will be using this machine for fx most of the time, then get him a node locked (which is cheaper). They allow to transfer node locked licenses to other machines if needed but not in an unlimited way of course. So for us we do have 1 floating and few node locked ones for people who are always working / using those workstations for the same stuff. For others we just get rentals for the durations that we need them.

I am sure Solaris has its way. I just think they made it too unfriendly to general user. Aovs = problem, usd assembly = problem, rendering on deadline = problem. Again at a large studio (which is what it's made for) not a problem at all. They have their own submitter, manager, TD's who can write stuff.

With all that said, I really like houdini. Yes it has a lot of quirks and especially in vray in houdini has its own logic, like having a frame buffer per rop rather than just one frame buffer like in max or maya. Having to duplicate sop inputs and merge objects into them (so essentially copying modified geometry) to create render layers, rather than modifying render override like in maya for example.

[Joelaff]

One more question because the SideFx support folks really don’t know much about vray…

When you do the export to vrscene for rendering on deadline can you export one file per frame so that the other render nodes can start the image rendering while the export is still going on? You can do this in Max and it more than makes up for the dealing with larger (overall for all frames) vrscene files.

[racoonart]

Just to add a few cents to the discussion... for what it's worth...

As has been said, solaris is complex. Way too complex for the amount of time a lot of people probably want to spend on it. I spent about 2 months learning it until I felt comfortable to rely on it. Technically, you *can* only use the more simple parts of it for scene assembly and management but in my opinion you only know what what that is if you know way more about solaris than you probably want to. As is often the case with houdini, solaris makes easy things hard and hard things easy. Personally, I really like it but it took a while to get used to it. That said, rendering via deadline is a huge pita and I'm really pissed about amazon not giving a crap. There are people who made their own submitter and put it online in a thread on the thinkbox forum but it's a huge mess since no one from amazon can be arsed to confirm if it's ok to put it on github for people to work on - so yeah, big drawback.

Regarding vray itself, we would like to use it, we've been using vray in max forever, but I'm hesitant. First of all, we're still rocking perpetual licenses and henceforth have no access to houdini vray without buying into subscriptions... and why would we give up our perpetual floating ones. And then there's the elephant in the room. Chaos is now an investment company optimizing for profit. Vray for houdini can't be profitable, just look at this forum here and how many distinct users there are. I doubt vray for houdini supports itself, let alone makes a profit - and I somehow don't think that's a good thing for the future of the plugin. I currently cannot, in good conscience, put my trust and long term productions into something I have no faith in being available in a year or two. Who knows what will happen, but it's too much of a gamble for me at this time.

[Morbid Angel]

One more question because the SideFx support folks really don’t know much about vray…

When you do the export to vrscene for rendering on deadline can you export one file per frame so that the other render nodes can start the image rendering while the export is still going on? You can do this in Max and it more than makes up for the dealing with larger (overall for all frames) vrscene files.

Vray does allow output of vrscene per frame. Which is what we use. We use variables to specify the output and we treat the vrscene output just like any render layer. Once those frames become available in vrscene, then the render frames pickup, so it does not way on per job basis, rather per frame. With that said, for us in 90% of the cases it writes the vrscenes so fast I just batch them, so for example I have 4 machines doing 100 frames so its 100/4 (25 frames per machine) and the writing is usually under few minutes.

If I did have a scene where I had large vrscene writes we're talking hundreds of megs or even gbs, then it makes sense to do few frames per machine. In this case more houdini engine/vray render node licenses would be better since you will speed up the export process. It's an additional overhead layer max/maya people can never care for if they don't use standalone and I miss those times...

[Morbid Angel]

Regarding vray itself, we would like to use it, we've been using vray in max forever, but I'm hesitant. First of all, we're still rocking perpetual licenses and henceforth have no access to houdini vray without buying into subscriptions... and why would we give up our perpetual floating ones. And then there's the elephant in the room. Chaos is now an investment company optimizing for profit. Vray for houdini can't be profitable, just look at this forum here and how many distinct users there are. I doubt vray for houdini supports itself, let alone makes a profit - and I somehow don't think that's a good thing for the future of the plugin. I currently cannot, in good conscience, put my trust and long term productions into something I have no faith in being available in a year or two. Who knows what will happen, but it's too much of a gamble for me at this time.

I would agree with you few years ago, however we've used vray for houdini in production for last 8 month with few issues. No more issues than in maya/max. Yes there are some rough edges here and there for sure. I believe however vfh is being actively developed right now, I have been working with the devs on few issues recently and they are getting/got resolved. I think so long as vray exists in general vfh is a strong candidate as I don't see houdini going away anywhere. If you have vray license already you can give this a shot. However with that said, if you are already setup with karma or arnold, those are strong competitors.

No one knows what the future holds though, I never thought I'd give up my beloved maya, but guess what autodesk greed kills small studios and they couldn't care any less for our success.

[racoonart]

Oh I'm not saying the quality of the plugin isn't as good as the others, quite the contrary. It's probably better because there are less people bothering the developers day in day out.... so you get your fix quickly. It's only about chaos having different goals now. I've used vray for houdini years ago and it was good, but I just can't trust chaos anymore. They won't axe vray for max anytime soon... but with houdini I'm not so confident.

[Joelaff]

Thank you both. Great info.

racoonart what renderer are you looking to be using with Houdini then?

Of course with Arnold you are back to Autodesk again.

[racoonart]

racoonart what renderer are you looking to be using with Houdini then?
Of course with Arnold you are back to Autodesk again.

Yeah, that's the big question, isn't it? Redshift used to be good, but apparently it has gone downhill a bit since maxon bought it. Arnold? Ha... no thanks. The renderer certainly is good but the company isn't. Renderman supposedly is good, but it's not cheap and I don't have personal experience with it (yet). Can't be too bad if ILM uses it, but it definitely depends on the kind of jobs you do. Personally, I'm currently using Karma - it's no vray, it's missing a lot of stuff... but it also is a lot better than I thought. It can't replace vray in advertisement and archviz and stuff like that, but for animation and fx work it's something to keep an eye on. It certainly surprised me. As a studio we won't be using it for most jobs but I'm slowly doing more and more with it when there's a chance for it.

But yeah, there's not that much choice realistically. If you need lightcache for quick secondaries, there's only vray or redshift.

[Joelaff]

I thought Maxon was supposedly a good company; are they going the way of Autodesk? Don't get me started on The Foundry's pricing, and horrendously slow, mainly single threaded Nuke.

[racoonart]

Maxon is part of the nemetschek group, essentially the same kind of corp as autodesk. According to threads I've read in the sidefx forums, people are not too happy about the progress on redshift since they sold out. I used redshift before that and cut all ties after. It was good back then, one of the most production ready renderers for H - now I hear people switching to karma from redshift, which I found surprising. Yeah, let's not get into foundry , they can do whatever they want without consequences due to lack of serious competition and they know it.