Distributed render over VPN

[alexnode]

Hi all,
I am using a Hamachi VPN to render over my rendering farm from a remote location. My file is is 147mb at the moment and it can grow. It's a building.
Unfortunately it takes 6 minutes to transfer the file to my rendering farm over the internet. Even if I change just the lighting intensity for one light it seems that the whole scene is transferred every time. I have local cache enabled. Do I do anything wrong or is that how it works?

Thank you in advance

[Edit] I use vray 3.30.05 I don't have textures on the model yet, and my upload speed is approximately 20mbit/sec that's about 150 MB per minute (not great but I was expecting one or two minutes of wait not 6) my model is 147 mb. My material library is cloned in all my machines via dropbox so I don't need to transfer missing assets. But it takes 6 minutes to begin the render. The machines are 3 x dual 14 cores xeon v3 @2.6ghz(I think?) plus a couple of i7 4930k at 4.2ghz, which they can load things really fast.
I used BF-BF to make it easier to see when it starts. When I tried with a few boxes and teapots it starts immediately. So my question is when I change a material or a light do i need to resend the whole model every time or Vray tracks the changes and keeps the model in the cache ? My impression was that If I check used cached assets, uncheck restart servers on end and transfer missing assets that I won't need to send the whole model everytime.

Apologies for not having a clue about vpn's and distributed rendering but it is the first time I am attempting to work like that. I need to be to another office for a couple of weeks to design stuff with an external team and I was hoping that I could use my servers at our place. You never think about things like that with gigabit ethernet. Do you have any ideas how can I imporve the 6 minute wait. I want to finetune materials and I was hoping for to be able to work a bit more quickly.

[vlado]

You could X-Ref to another file all the geometry and other stuff that you won't be changing, and leave the lights and the camera in the master file. Then only the smaller master file will be transferred each time you do DR.

Oh and make sure compression on Max files is turned on.

Best regards,
Vlado

[alexnode]

This worked really well vlado. There must be some problem with the vpn because through dropbox I can synchronise the 150mb files in a few seconds.
I can work quickly again .

[vlado]

Things are generally slower when they go through VPN as there is a lot of extra overhead involved. Still, 6 minutes sounds a bit too much for 147 megabytes. I have to try this at home...

Best regards,
Vlado

[benb32]

This worked really well vlado. There must be some problem with the vpn because through dropbox I can synchronise the 150mb files in a few seconds.
I can work quickly again .

Do remember that dropbox uses block level remote differential compression. So if you are updating a file, only the parts that have changed are actually uploaded. That's why it hammers your CPU when windows starts up - checking for any changes where the local and remote timestamps are different.

[alexnode]

Do remember that dropbox uses block level remote differential compression. So if you are updating a file, only the parts that have changed are actually uploaded. That's why it hammers your CPU when windows starts up - checking for any changes where the local and remote timestamps are different.

For that case though it works really well, in a few seconds the files travel 25 miles! I was in an office once upon a time that we were using ascii maya files with SVN (Apache Subversion) and it was working really well for updating and synchronising only the parts we were working with. (We were using point cloud 3d scanning material, A lot of gigabytes per file). But it seems a bit of a pain to install ... if you are IT illiterate .
I am happy with dropbox to be honest ... even though everyone else complains about it.

[benb32]

In terms of features and reliability dropbox comes out on top in cloud sharing - don't even look at Onedrive for business, google drive and Onedrive personal are alright, but don't use differential compression with files other than office productivity. Versioning (done on a file level, not folder) is better with Dropbox, and as a turnkey setup with no difficult setup is perfect.

There are a few other solutions that have come from the Linux rsync idea that work incredibly well with syncing desperate large files between sites. Let me know if you want any more info.

[alexnode]

I have a pro google drive and onedrive from office, but I was forced to get them because a client was using them. I haven't done any work there apart from adding renders and text documents, I like the comments and collaborative writing, but you know ... I make pictures ! I think that I might find it difficult to use Linux, I have an ubuntu partition on a laptop but because of 3ds max I am bound to windows until I quit We don't have IT because we can't afford it ! ... or we prefer buying rendering nodes.

[benb32]

There are a few non complicated solutions you can use in windows . Don't have to use Linux at all, just mentioned that's where the implementations first appeared, due to the use of perl or java.

Sounds like using keeping your locations within a Dropbox folder would be the quickest and easiest option. You wouldn't even need a VPN, except to initiate the render.

[alexnode]

Sounds really interesting, what do you have in mind ?

[benb32]

Thinking the idea to keep your assets, proxies and Xrefs based in a known location within a Dropbox folder that you and your nodes can see. Then you have a much more efficient way of syncing the resources amongst the machines. New stuff would be compressed and copied (not great with already compressed pictures, but a lower overhead that vpn), changed things would transfer really well.

Bear in mind that you wouldn't want your central Max folder to be on Dropbox as the multiple small changes would generate too much traffic. Would also do tests with whether to use compression when saving your Max scenes. It might be that when you do copy your Max file to dropbox, it will do it more quickly than max compressing and then dropbox transferring. Scene compression is not that quick, especially on files larger than 500MB - just look at how autosave effects Max. The whole speed of things is more closely related to single threaded speed of your CPU... i.e. a core clocked at 4.5 GHz appears to save more quickly than one clocked at 3 GHz - but granted we are only talking about a few seconds more, unless you're using a PCIe nvme solid state hard drive.

[benb32]

http://web.synametrics.com/Syncrify.htm

I know this is presented as a private cloud backup solution, but when you enable the server side file cache, it's one of the most efficient solutions I've used to keep multiple locations synchronised. It's fully windows compatible, just install and configure and it works.

I've used it to keep hundreds of GBs synced on a nightly basis even without fibre broadband...