Auto Convert to TX

[tdugard]

Hello everyone,

I'm not here to discuss the benefits of using TX file format for rendering. But to make Vray more user friendly when we use this file format.

I wrote custom scripts for all artists to help them convert all their textures to TX format (using make tx) and manage them in a 3dsmax scene.

Actually i use a custom script :
- If there is no TX, the script builds one and replace the path in the VrayHDRI/VrayBitmap node filepath.
- If there is a TX, the script compares dates between the original map and the tx one and if the original map is newer the script will rebuild the TX and replace the path in the VrayHDRI/VrayBitmap) node filepath.
- The script store in an AppData for each VrayHDRI node the original complete path of the texture to revert easily to the original texture (and to backup the original file with the tx).

But this workflow is really annoying when you need to work on the texture :
- Once the TX is loaded in the VrayHDRI/VrayBitmap, you can't work on photoshop for example, save your image and have directly the result in 3dsmax, you need to manually convert the file to tx to see the result. When you do it 5 or 6 times in 10 minutes it's very annoying. Of course, you can switch back to the original format and once you're done convert it back to tx.

So what i would like to have in vray is a option to automatically convert to tx all the newest textures (checked by date) in the tiled textures option but keeping original path in VrayHDRI (TX is just a render file format). And also add a checkbox in the VrayHDRI (VrayBitmap) if you don't want to use the TX file format for this specific map, i've made a mockup :


Of course, the first time it will be longer, but after the first conversion, only the newest textures are converted so the process is really fast.

It can really help the artist to be more efficient so he can keep his original format to do back and fourth while working and use the power of tx files while rendering without having to convert manually. (You can find this option on Arnold and it's really useful)

Thanks for your time.

[^Lele^]

(You can find this option on Arnold and it's really useful)

There is also something similar in Redshift.
Of either implementation, i loathe the utter messiness they promote in workflow.
There is no clear place the tiled textures go (they're not side-by-side. in the case of RS they get also a name conversion to some hashed string.), nor it's ever clear to the user if the process has worked fine or not.

Try it in Arnold.
Check Auto-mip and Auto-tile, and Render.
This ought to have converted the textures.
Then uncheck those, and also uncheck "accept unmipped" and "accept untiled".
Finally check "Use Existing .tx Textures".
This, in essence should tell Arnold to render with the just-converted TX files.
Render, it won't.
Where's the .TX now? How can the user verify it worked as intended? Why's the render not starting if the TX was converted? (Hint, it never was, in max.)

Also, either implementation would have serious troubles with DR, if they had the feature.

I have been writing tools for tiled conversion in production before, and they were *always* tied to properly structured workflow.
In other words, conversion to tiled is done once the asset is out of lookdev, and ready to be dropped in sequences.
This is entirely analogous to converting to proxy models (i.e. render-specific cache formats) being actively worked on.
Doing the conversion for every stroke of paint added to a texture (or every pulled vertex of a model) is not only impractical, it is needlessly costly and time consuming, on top of being prone to confusion and mistakes.

Having some form of implicit conversion, in my opinion, only hampers workflow, and promotes the messier approaches.

I find much more elegant to run a pre-render phase and do the checks and necessary conversions with your scripts (if you can't be bothered to explain workflow needs to artists, write a sanity checker, right?), so you'll have a handle on proceedings, than leaving it to a third party which will make the choices they'll need to make it work universally, as opposed to your specific case.

Devs may disagree with my opinion, of course.

[vlado]

We've been having some discussions on easier workflow with .tx files; hopefully we will manage to make it more user-friendly. Thanks for the suggestions, we will definitely consider various options

Best regards,
Vlado

[Dart_Design]

I wrote custom scripts for all artists to help them convert all their textures to TX format (using make tx) and manage them in a 3dsmax scene.

Actually i use a custom script :
- If there is no TX, the script builds one and replace the path in the VrayHDRI/VrayBitmap node filepath.
- If there is a TX, the script compares dates between the original map and the tx one and if the original map is newer the script will rebuild the TX and replace the path in the VrayHDRI/VrayBitmap) node filepath.
- The script store in an AppData for each VrayHDRI node the original complete path of the texture to revert easily to the original texture (and to backup the original file with the tx).

Hello tdugard
I am totally agree with you. We are missing exactly this options which you described.

You told that you made some custom scripts for all artists, can you please tell me, where can I download them? I am dealing with a problem where I need to revert all my textures to original formats. They are all saved in their original folders, just near TX textures but there is no option to revert...

Many thanks in advance



Hello vlado
Right now we can only convert all textures of the scene at once and without any revert option. Looks like all or nothing without step back (at least, for common user).

It will be also great if we had this options in Vray converter:
- convert textures of selected objects to tx
- revert to original file format (all objects or for selection)

Thanks in advance

[^Lele^]

Right now we can only convert all textures of the scene at once and without any revert option. Looks like all or nothing without step back (at least, for common user).

Would you care to show me a sample from any application able to do this?

It will be also great if we had this options in Vray converter:
- convert textures of selected objects to tx

This is being worked on for the V.6 cycle.

- revert to original file format (all objects or for selection)

This is a logical impossibility for a few reasons: there is no trace of the original texture file anywhere in the scene once one swapped the paths out.
The entropy can't be undone unless one kept a separate list of what was what.
That list, in general, is stored in the unconverted max scene by the user.

Trying to backpedal to the original format without said list of assets can only work if the textures have been converted side-by-side with the original ones (not a given.), and if the original textures are exactly one per .tx (not a given. people may store layered TIFF files and flattened PNGs in the same folder. The converter wouldn't be able to choose based on filename alone.).

So, the way out is to save the scene before conversion, converting, and then in case re-import the old materials and nodes to revert the conversion.

[Dart_Design]

Would you care to show me a sample from any application able to do this?

I don't know. I am not software expert, but I see what functions we are currently missing. That's why some users are starting to develop their own scripts.

For example, V-Ray Material Library to Regular Texture from Mohammadreza Mohseni.
https://mohseni.gumroad.com/l/qjOBB

this is simple tx->tiff conversion script. I assume, batch converting tx -> jpg or tiff -> jpg with auto replacement of texture file format for paths is also possible.

What for now tx converter in vray does, it creates a tx copy in the same folder and with the same name. We simply miss at least the reversion functionality to originally stored textures for the whole scene or for selection only.. But for sure, the tx->tiff or jpg conversion would be also great to have.

Of course, such solution may not work for all cases. But this info can be delivered via simple warning window. Something like " bla bla bla if you have advanced layered textures, their converted copies will be flattened"

[^Lele^]

I don't know. I am not software expert, but I see what functions we are currently missing.

We do too, but we also consider why that is, and what it would take to implement them.

That's why some users are starting to develop their own scripts.

This is a great attitude in general.

For example, V-Ray Material Library to Regular Texture from Mohammadreza Mohseni.

I tried this, but it downloads a zip with an exe in it, and the exe is "password protected". Unfortunately no password was forthcoming.
It is also heuristically detected as a generic malware by malwarebytes.
I hope the script doesn't just rename the .tx files, however, as that would mean wasting users 50% of the texture file size, for no benefit whatsoever (that's because the mipmaps sorted in the file aren't discarded.).
The right way to flatten the mipmaps would be to open in a third party app and resave as a new format.
EDIT: the author answered and passed me the "script", which is unfortunately just a .bat file that renames all textures in the folder to .tiff, in the process breaking the material library.

this is simple tx->tiff conversion script. I assume, batch converting tx -> jpg or tiff -> jpg with auto replacement of texture file format for paths is also possible.

Sure. There's photoshop for this, though: .tx files *are* tiffs and can be opened as such (either with "open as..." or via previous extension renaming.).
You wanted to *revert* a *converted* scene, and that isn't possible in a universal fashion, as explained above.

What for now tx converter in vray does, it creates a tx copy in the same folder and with the same name. We simply miss at least the reversion functionality to originally stored textures for the whole scene or for selection only.. But for sure, the tx->tiff or jpg conversion would be also great to have.

Yes, that's a wise way to convert textures, but unfortunately we cannot control what the user does.
And moving them elsewhere with repathing is all too common an occurrence, thereby breaking the relationship, as mentioned in my previous post.
See above for conversion to tiff.

Of course, such solution may not work for all cases. But this info can be delivered via simple warning window. Something like " bla bla bla if you have advanced layered textures, their converted copies will be flattened"

We do not look into file contents. And even if we did, we wouldn't be able to back-swap a texture that was moved, or one for which there were more than one version in the same folder (mytexture.tif, mytexture.png, mytexture.tx)
A tidy pipeline is the user's task, i'm afraid, as we can't infer intention.

[Dart_Design]

lele

okay. I understand your position. As your disclaimer says :The views and opinions expressed here are my own and do not represent those of Chaos Group, unless otherwise stated.
Can you please tell me, is it your private view and opinion or this can be seen as official chaos group answers ?

This is being worked on for the V.6 cycle.

I have checked on version 6. There is no changes at all.

I tried this, but it downloads a zip with an exe in it, and the exe is "password protected". Unfortunately no password was forthcoming.

Please, look at the password on the main page. The text is marked bold.

Would you care to show me a sample from any application able to do this?

Yes, I have found some script called Pixamoon BTR.
It can convert TX for example to jpg with a lot of options. It can also change the internal file format from tx to jpg, so you can very simple revert the tx to original format, which vray converter can not. (and will not ?)

If chaos group will not/don't want to implement some revert options to TX converter, then please, add the WARNING message that this operation can not be undone and it is good to make a copy of original file, because now we have the situation "ALL OR NOTHING WITHOUT STEP BACK"


... as tdugard wrote, his script can store in an AppData for each VrayHDRI node the original complete path of the texture to revert easily to the original texture. Why even this not possible to implement in build-in converter ?

You can find the screenshots of Pixamoon script in my attachments.

[^Lele^]

lele

okay. I understand your position. As your disclaimer says :The views and opinions expressed here are my own and do not represent those of Chaos Group, unless otherwise stated.
Can you please tell me, is it your private view and opinion or this can be seen as official chaos group answers ?

My technical opinion is more informed than, say, my opinion on how good the weather is on a given day, the latter being what the disclaimer tendentially refers to.
Specifically, i wrote the code to convert to .tx, and it can possibly be expanded to convert all textures to some format like PNG, and EXR for higher range images, and re-path them.
However, as there are options within the standard array of user toolsets (save the file before conversion, for example, so to merge/revert to it later as needed. And Photoshop or equivalent for batch image conversion.), and the fact that it'd end up duplicating an otherwise perfectly valid set of textures (the ones previously converted to .tx, which we established can't be reliably reverted to.), i don't think it'll get a lot of traction.

I have checked on version 6. There is no changes at all.

I used the word "cycle", not "6.0 as released".
Work is ongoing.

Please, look at the password on the main page. The text is marked bold.

I spoke directly to the author that passed me the .bat file that duplicates and renames the textures on disk without re-pathing anything, so f.e. grabbing the library materials still creates them with .tx files referenced.
Alas, you do see the tiff thumbnails in windows explorer.
This can be done via batch files, or with any of the myriad free file renamers.

Yes, I have found some script called Pixamoon BTR.
It can convert TX for example to jpg with a lot of options. It can also change the internal file format from tx to jpg, so you can very simple revert the tx to original format, which vray converter can not. (and will not ?)

I know Pixamoon scripts quite well: it's a complicated thing to write properly.
Simple and straightforward workflow ideas tend to get awfully messy once deployed in the real world.
There are very good reasons why Pixamoon sells it for 18 dollars.
Anything i may write would not be that sophisticated.
You also have makeTX.exe in the bin folder of V-Ray, should you want to convert your .tx files to some other format regardless of any effort on my part.

If chaos group will not/don't want to implement some revert options to TX converter, then please, add the WARNING message that this operation can not be undone and it is good to make a copy of original file, because now we have the situation "ALL OR NOTHING WITHOUT STEP BACK"

As mentioned, save the original Max file, convert, save the max file to a new name: i'm not sure this needs a warning, as it's general good practice before massive scene operations, and the script doesn't save the file for you, much less so overwriting the original one.
We can surely add an undo entry if you prefer, but it's generally more troublesome with this kind of operation (as opposed, f.e. to changing a keyframe value and reverting it back.).

... as tdugard wrote, his script can store in an AppData for each VrayHDRI node the original complete path of the texture to revert easily to the original texture. Why even this not possible to implement in build-in converter ?
You can find the screenshots of Pixamoon script in my attachments.

There are two functionalities you are asking for interchangeably, but they are not equivalent:
a) Scene reversion to the original textures. That is a hard pass for me. Save the file before conversion, revert/merge later as needed. File names change, external file-stored connections break.
b) File conversion from .TX to some other format and re-pathing. This is definitely doable, but it too has issues, and it is quite likely to get very complicated to write properly for the real world.
I will look into it, but i'm making no promise, other that it won't be anywhere near as complete as that of Pixamoon for features, should i ever get to writing it.

[^Lele^]

I spent a few days looking into this, and unfortunately it doesn't look very good.
What Pixamoon does is not only clever, but also very custom: he embeds his own .tx reader inside the script, and runs it to get the .tx files' preview and info, otherwise impossible to retrieve via simple maxscript methods, or invoking makeTX.exe.
In practice, mimicking even the basic functionalities of the script requires quite the effort from the part of devs, on top of any scripting i may bring to bear.
I'll continue to poke at the issue, but you're much better off using the mentioned alternative methods, as a solution isn't forthcoming in short order.

[tdugard]

Would you care to show me a sample from any application able to do this?

This is a logical impossibility for a few reasons: there is no trace of the original texture file anywhere in the scene once one swapped the paths out.

Arnold is doing it right out of the box, You put your original path in the bitmap texture node (jpg, exr, tiff) you check Auto convert to TX in the render engine properties and the engine will do the conversion on the fly. If you make some modification on the original texture you just have to save it and since the date is newer than the tx, it will be regenerated again. It's the simple way of doing it without having to store the original map or path because you are working with it you never manipulate tx (everything is done under the hood).
We just need a checkbox inside the VrayBitmap to prevent the auto conversion for some specific maps if we need too.

For me it doesn't seem to be that complicated, i could write it using pre render callback to switch everything on the fly and after the render put it back to the original map but the time to swap the path and max to reload is way too long.

I've attached the way it work in arnold. (the render.tx map is created next to the original bitmap and the render uses it instead

Tony.

[^Lele^]

I am well aware of how both Arnold and Redshift do it, as i replied in post #2.
Without explicit file paths, the method is rife with potential issues.
You can of course write whatever you'd like to, so long as it fits your specific pipeline's needs.
We have DR and a few other things to consider, that neither of the above supports out of the box (and i suppose won't support without explicit file paths either, regardless. See TeamRender in c4d.).

There may be a more elegant way about the conversion and reversion to TX, but it's not clear to me yet if it'd work universally.
All this said, something will have to happen at some point, and the conceptual issues will have to be resolved or simply fail.

[tdugard]

The problem is that i can't write those callbacks scripts because changing the path manually before and after render make 3dsmax to hang bacause of the need of reloading all textures that's not an option.

In post #2 you mention that is analogous to the proxy workflow, you are right but with proxys you can easily revert back to the original model if you need it.

I'm not telling that Arnold feature is a bug free option, but it's really a nice feature and i'm sure it can be a real benefit in everyone's pipeline if something similar is implemented.

And you said :

Try it in Arnold.
Check Auto-mip and Auto-tile, and Render.
This ought to have converted the textures.
Then uncheck those, and also uncheck "accept unmipped" and "accept untiled".
Finally check "Use Existing .tx Textures".
This, in essence should tell Arnold to render with the just-converted TX files.
Render, it won't.
Where's the .TX now? How can the user verify it worked as intended? Why's the render not starting if the TX was converted? (Hint, it never was, in max.)

I've tried in max 2023 and it work as expected. It renders properly and tx file is still there.

With the DR it should just be a check before the render started on those DR machine, the main worker from where you start the distributed rendering makes all the tx conversions or better the engine split all the needed convertion on all worker before they started to render.

[^Lele^]

Yeah, Arnold works now.

The thing that bothers me endlessly is that even side-by-side conversion would not be an option in some pipelines: the lighter may have zero rights to write in the textures folder, for example.
So the client that did the auto conversion to .tx would have to specify a different path, and then we may have name clashing and so on (imagine textures uniquified also via folder structure. If the folder structure disappears, one gets name clashing.).
That folder may then not be shared among the other render boxes...

Cut it as we may, it looks like we just do not have enough information retention in the process, which makes it quite risky to approach for unknown, idealised pipelines.
A risk at some point we'll have to take, i guess.

On callbacks, if all you do is check for new texture versions to reconvert to .tx, there should be no pathing change at all, so long as the scene has .tx files plugged.
But perhaps i haven't understood the scope of your request.

[tdugard]

On callbacks, if all you do is check for new texture versions to reconvert to .tx, there should be no pathing change at all, so long as the scene has .tx files plugged.

The problem is for the artist to keep their original textures plugged in not the tx file format only used at render time. So the script has to put the original path once the render is over.

The thing that bothers me endlessly is that even side-by-side conversion would not be an option in some pipelines: the lighter may have zero rights to write in the textures folder, for example.
So the client that did the auto conversion to .tx would have to specify a different path, and then we may have name clashing and so on (imagine textures uniquified also via folder structure. If the folder structure disappears, one gets name clashing.).
That folder may then not be shared among the other render boxes...

See that as an option, user may put this on or not. Some may need to convert tx themselves like before but other may benefit a lot from this feature. The path can be overided on the render properties (even with the use of some variables like we see on timestamp in the vfb). Name clashing is a problem that can be solved i'm sure. Those folders problems can already be seen : the render nodes may not have the right to access the texture folder of the asset. This is an IT problem.

I see the TX process like the python __pycache__ system. Once you run a python code a bytecode compiled version of your scripts/modules is generated for speed purpose.

For me the coolest option is a .tx folder inside the texture path with all the tx inside. And an option in vray to force recompute the tx if something goes wrong.