We work on reasonably high powered workstations, opening all our work files, assets, textures, PSDs, models etc from our central server. All files are saved back to this server. We also have 10+ render machines doing mainly DR, but also BB jobs from time to time.
I believe that our workflow could be improved markedly if we were opening all of our files locally, and saving them locally too. Also, if all the render machines had local copies of our ASSETS drive, they should function more quickly too.
The problem with working locally, of course, is that our daily backups only backup our central server. If we work locally, things don’t get backed up.
What I would like to have is a local clone of the ASSETS drive on each and every machine (workstations and render machines). When rendering, textures etc will be pulled from this local clone. If we create new assets, or edit an asset, this files should be cloned back to the central server ASSET drive so that they will be backed up. All other machines on the network would need to constantly ‘watch’ this folder, syncronising when changes are detected.
I have all my texture assets, textures and proxies that I use all the time in every slave…I have a map folder inside of every project where I put all the modified or custom textures for that specific project … If I start to use those textures pretty often I copy them inside my asset drive under my company name( I have the textures by manufacturer as well as models).
I like to keep my textures that I purchase untouched, so updating the slave’s drive is not difficult…(Only new stuff or the custom folder)
So if my server ASSETS drive is, for example, the ‘x’ drive, do I just put a drive in each machine and make sure it is ‘x’ also?
How do I ensure that each machine is getting the maps from their local copy rather than from the server copy? Because we run spawner and backburner server as services, all maps/assets/ies/xrefs data etc needs to have UNC naming, so that is forcing them to look at the server.
i have the textures synced from server to local machine on some local harddisk - it doesnt matter where they are stored local, because i map the path (for example “D:/texture”) to “//127.0.0.1/textures/”
so i have on every machine same UNC-path and stored local
How do I tell a local drive to be seen as 127.0.0.1? (using windows 7, windows xp and windows sbs 2003)
Clearly I need to look into this as it seems that it is very possible to optimise things. It would be so much easier in a one-man-band, but with numerous staff and people working on different parts of the same project, things soon get complex.
Unless you are working on a Linux env where viable pipelines can be created natively, I suggest you avoid 3rd party applications and drive cloning and invest in improving your network instead. 10Gbps can help as can Raid6 and drive selection. Much better to keep things simple imo.
I copy all my assets up to the network, and then I copy from the network down to the slaves. I have MAX looking to a local drive for the maps, so each MAX is looking to C:\. I copy to the network so it can be grabbed by our nightly backups, other than that, I could skip that part. The process takes a minute and is automated with SyncBackPro.
I use allsync to syng all my textures to slaves… Its scanning constantly so all is fast. I basically tell all sync to copy my X drive to slave X drive every 1 min or so…
cause of synching files, i want to add something i wasnt sure
all of the bitmaps (only for shaders) and the whole materiallibrary are in the cloud, stored by some filehoster, synched in house by server. they get synched on the fly and gives us the possibility to add bitmaps from designer and graficstudios easy. so when you are on the way with a laptop, you are always up to date with a UNC-path localy.
Thanks all. So it seems we have a couple of options:-
1) Work the way we are, but invest in upgrading our network infrastructure to 10GB or fibre
2) Install a local drive (or 2) in each and every machine to replicate our ASSETS drive and WORK drive from the server, then make sure each and every machine is using localhost (127.0.0.1) addresses to point to textures/work files etc.
I would start with a local asset drive keeping the work files on the server initially…in my case, the local drive has the same letter as the main asset drive.
having a local work drive doesn’t work well for me because it’s one more thing that I have to worry about…I have enough sync my work drive in my laptop to the main when i’m working out the office.
We are set up with a local folder, manually synced over the network from our workstations using DSynchronize, that contains our most used and heaviest assets. This consists mostly of HDR/EXR images, proxies, and large textures. Our slaves also utilize a single 120 GB SSD each to maximize the speed that these assets are acquired, so we try to keep this folder clean and current. We did tests and found the nodes configured in this way to be markedly faster at producing an image.
I find that complex scenes sometimes result in DBR slaves only being able to actually start rendering WAY late into the render process on the main workstation. I guess this is due to a mixture of 1) max file transfer 2) maps file transfer 3) IRmap/LCache transfer. I might look into a super-fast network to get around this but I’m wondering if Vlado & Co. might have some tips for this, apart from what people are talking about here with localised copies (which I won’t be doing).
