Faster saving of IRRmaps

[Dynedain]

When you save an Irrmap (whether manually or automatically) it can take a very long time. For instance, right now I'm watching an irrmap save, and its writing at about 200kb per second. Even though it is writing to a network share, this is abismally slow and not even close to the throughput that is normally available for other operations.

Why does it take so long to save irrmaps? It makes people think that the rendering or max has crashed when dealing with large irrmaps.

[DaForce]

is it as slow when saving locally?

[Dynedain]

I'm not certain, but the irrmaps save incredibly slow over the network compared to any other type of filesystem access.

[DaForce]

Was there any other network traffic at the time? Or Was it just the saving IR map.

Also try saving the same IR map locally and see how quick it is. It may be some funny network thing.

[Dynedain]

Was there any other network traffic at the time? Or Was it just the saving IR map.

Also try saving the same IR map locally and see how quick it is. It may be some funny network thing.

It was only saving the irrmap. There was no other significant network traffic. This isn't a new phenomenom, or a one-time only phenomenom. I have noticed this since I began saving irrmaps all the way back on 1.09. Saving irradience maps takes much much longer than saving anything else out of Max, and much much longer than transfering files manually.

Saving the irrmap locally is almost instantaneous. The network connection is quite fast, and as I've mentioned, copying/transfering files manually happens at a normal reasonable rate. But for some reason, saving irrmaps, and only saving irrmaps, takes at least 10 times longer than anything else.

[DaForce]

SO its only copying IRmaps over networks. Other files are fine, and saving IRmaps locally are fine as well.
Which is odd cause i dont see how vray would handly the network saving any different than the local save.

Have other people reported this problem as well?
I might have to test it out myself.

[RErender]

There is a big difference between writing a file bit by bit to a network compared to copying the whole file at one time to the network. We had some scripts that were writing files item by item over the network into xml files, it took at least twnty times longer to create them on the network than to write them local and then copy them up. Some other scripts were sped up by compiling the all the data in local memory and writing to the network in one chunk. I suspect it is due to the server cpu needing to act as a "word proccesor" to write the data.

Maybe the issue is similar tor the imaps?

[Dynedain]

That sounds like it could be the fundamental issue. Lots of read/write operations will deffinately be slower than one chunk read/write operation.

That is something that I suspected. The question is whether saving Vray irrmaps can be done in larger chunks, thus speeding up the process?

[jacksc02]

i have to give this a bump.

But it seems the main issue is with saving the irmaps to a UNIX based server

[vlado]

V-Ray does indeed write the files in very small chunks (sample by sample), but in theory, this should be buffered into larger chunks by the OS... at least this is what happens for local files. It might be possible to rework this part so that the samples are written in groups rather than one by one.

Best regards,
Vlado

[jacksc02]

that would be great vlado, since upgrading to a UNIX server this has really killed us

[studioDIM]

What about, and i am very very hypothetical here, saving the maps within the max file itself?
I've had some success doing so (scripting), but reading from an already saved map, and recreating it on disk in one go.
That may solve some issue of this type, as the map would sit in memory and be saved as a single chunk with the max file.
A maxzip would help in case reducing the filesize, and comparatively, halve the irmap size over the network, reducing the load considerably.

Lele

[jacksc02]

sounds interesting lele, maybe we can work on it?

[studioDIM]

Not I.
That is, what i can do with scripting is embed it into dummies.
But the script has then to be rerun to recreate the map on disk, which could be lenghty, and error prone, as the handling of write buffers is somewhat clunky due the the way maxscript allocates memory.
Of course I don't know how easy/difficult this could be for Vlado to implement, even though i know fR had it from day one (and that's where i ripped the idea from...).
I can sure send you what i have right now, which does work (unless i did so already, it's been a while now...), even though it is of limited usefulness, and not entirely foolproof (renaming or deleting one dummy atm simply cuts a map bit out, which ain't ideal as the results could range from crashes to odd lighting upon rendering).

Lele

[Dynedain]

What lele has suggested is basically how Radiosity solutions are saved in Max and may not be a bad idea... on the other hand our irrmaps can get reaaaally big...

If Vlado is able to figure out a way to change how big the write packets are, that would be best. It's been discussed that for some reason, Max saving images or files, or even the vray image output doesn't suffer quite as badly from this problem as saving irrmaps does.