Local copies of server files

[tricky]

We work on reasonably high powered workstations, opening all our work files, assets, textures, PSDs, models etc from our central server. All files are saved back to this server. We also have 10+ render machines doing mainly DR, but also BB jobs from time to time.

I believe that our workflow could be improved markedly if we were opening all of our files locally, and saving them locally too. Also, if all the render machines had local copies of our ASSETS drive, they should function more quickly too.

The problem with working locally, of course, is that our daily backups only backup our central server. If we work locally, things don't get backed up.

What I would like to have is a local clone of the ASSETS drive on each and every machine (workstations and render machines). When rendering, textures etc will be pulled from this local clone. If we create new assets, or edit an asset, this files should be cloned back to the central server ASSET drive so that they will be backed up. All other machines on the network would need to constantly 'watch' this folder, syncronising when changes are detected.

Is this kind of setup possible?

[glorybound]

Well, that's what I do. I use SyncBackPro. I work locally and sync my slaves, so everything is mapped to a local drive.

[flino2004]

I have all my texture assets, textures and proxies that I use all the time in every slave....I have a map folder inside of every project where I put all the modified or custom textures for that specific project .... If I start to use those textures pretty often I copy them inside my asset drive under my company name( I have the textures by manufacturer as well as models).
I like to keep my textures that I purchase untouched, so updating the slave's drive is not difficult...(Only new stuff or the custom folder)

[tricky]

Well, that's what I do. I use SyncBackPro. I work locally and sync my slaves, so everything is mapped to a local drive.

So if my server ASSETS drive is, for example, the 'x' drive, do I just put a drive in each machine and make sure it is 'x' also?

How do I ensure that each machine is getting the maps from their local copy rather than from the server copy? Because we run spawner and backburner server as services, all maps/assets/ies/xrefs data etc needs to have UNC naming, so that is forcing them to look at the server.

[mario74]

i have the textures synced from server to local machine on some local harddisk - it doesnt matter where they are stored local, because i map the path (for example "D:/texture") to "//127.0.0.1/textures/"

so i have on every machine same UNC-path and stored local

[tricky]

i have the textures synced from server to local machine on some local harddisk - it doesnt matter where they are stored local, because i map the path (for example "D:/texture") to "//127.0.0.1/textures/"

so i have on every machine same UNC-path and stored local

[ATTACH=CONFIG]13524[/ATTACH]

How do I tell a local drive to be seen as 127.0.0.1? (using windows 7, windows xp and windows sbs 2003)

Clearly I need to look into this as it seems that it is very possible to optimise things. It would be so much easier in a one-man-band, but with numerous staff and people working on different parts of the same project, things soon get complex.

[deflix]

Unless you are working on a Linux env where viable pipelines can be created natively, I suggest you avoid 3rd party applications and drive cloning and invest in improving your network instead. 10Gbps can help as can Raid6 and drive selection. Much better to keep things simple imo.

[mario74]

of course, this is some aspect.


@ tricky: first you have to give access to local folder in windows, so you will see this folder on network

type in command-prompt: net use x: "\\127.0.0.1\my_local_folder" /persistent:yes

while x will be the name of drive you will get, persistent means this will be stored and available after reboot

for more information: http://www.cezeo.com/tips-and-tricks/net-use-command/

edit: grab textures from x than

[glorybound]

I copy all my assets up to the network, and then I copy from the network down to the slaves. I have MAX looking to a local drive for the maps, so each MAX is looking to C:\. I copy to the network so it can be grabbed by our nightly backups, other than that, I could skip that part. The process takes a minute and is automated with SyncBackPro.

[Dariusz Makowski (Dadal)]

I use allsync to syng all my textures to slaves... Its scanning constantly so all is fast. I basically tell all sync to copy my X drive to slave X drive every 1 min or so...

[mario74]

cause of synching files, i want to add something i wasnt sure

all of the bitmaps (only for shaders) and the whole materiallibrary are in the cloud, stored by some filehoster, synched in house by server. they get synched on the fly and gives us the possibility to add bitmaps from designer and graficstudios easy. so when you are on the way with a laptop, you are always up to date with a UNC-path localy.

proxys and ies are still stored in house

[tricky]

Thanks all. So it seems we have a couple of options:-

1) Work the way we are, but invest in upgrading our network infrastructure to 10GB or fibre
2) Install a local drive (or 2) in each and every machine to replicate our ASSETS drive and WORK drive from the server, then make sure each and every machine is using localhost (127.0.0.1) addresses to point to textures/work files etc.

[flino2004]

I would start with a local asset drive keeping the work files on the server initially....in my case, the local drive has the same letter as the main asset drive.
having a local work drive doesn't work well for me because it's one more thing that I have to worry about.....I have enough sync my work drive in my laptop to the main when i'm working out the office.

Fernando

[mario74]

that is a point of costs now. speeding up your network without a RAID - and you will have almost the same speed as now!

you cant mess with local data and backup them during downtime. that small jpg's or png's to sync on the fly you cant feel.

[beestee]

We are set up with a local folder, manually synced over the network from our workstations using DSynchronize, that contains our most used and heaviest assets. This consists mostly of HDR/EXR images, proxies, and large textures. Our slaves also utilize a single 120 GB SSD each to maximize the speed that these assets are acquired, so we try to keep this folder clean and current. We did tests and found the nodes configured in this way to be markedly faster at producing an image.