Import/Export Paradox mesh tool

[Sid Meier]

@Sid Meier - yep you figured it out, you have to delete the material for it to not appear in the "Set Mesh Order" popup. (that dropdown just lists all PDX materials in the scene)

I don't think having the extra material left over should cause problems with export though, as it'll just get skipped/ignored. But I'll do some testing to be sure.
Deleting a bone is definitely a problem though.

Basically suppose I had two Hair meshes; a_hairback and z_hairfront.

I had these two meshes rigged; but the rig was quite clunky, since I had 4 different "hair styles" and had 1 rig to cover all of them; and I was encountering an issue where the hair animations (and only the hair) were experiencing some major flickering I couldn't quite understand; it was too quick to really know if it was a texturing issue, a deformation issue, etc.

So I deleted the hair bones, then the hair meshes; made *four* separate sets of HairFront and HairBack (Hair1_Front, Hair1_Back, Hair2_Front, Hair2_Back, etc) and four sets of rigs for them (I am currently past the 50 joint limit and now I need to go back and delete 7 bones which is quite annoying, there's no reason not to have 100 bones or 1000 bones as the limit tbh) such that each can be rigged an animated separately. (Basically I use a trick with the UV mappings; I separate the hair textures to four corners of my texture file, so the Hair1 texture will show but the other 3 textures will be transparent/invisible).

However when I go to export, I get the following error: "A vertex was mapped to bone Hair_Lower1.R that is no longer present..." or something along those lines.

I made sure that for every visible mesh no deleted vertex groups remained and re-normalized all. But clicking on "Set Mesh Order" I noticed that a_HairBack and z_HairFront remained in the list.

After figuring out how to delete the materials then I export worked (I had switched every "hair" mesh to a duplicate material except for the ones that needed deleting).

So deleting the material was a work around to the deleting the bone/mesh issue to be clear. If I delete a mesh that has an material attached; it should probably automatically remove that mesh from the export list.

 

[StarryWisdom]

Symmetrical Normal Map Issues...


I’m having problems with my normal map for my slipfighter. The right side, 1st picture appears correct but the left side, 2nd picture, looks to be doing the opposite it is supposed to do.





I’m using blender, the model was at one time mirrored using the Mirror Modifier but was applied and the right side UVs were offset 1 quadrant over to reduce baking issues before being exported to 3D-Coat.



I exporting my high resolution and low resolution model to 3D-Coat, where I baked it and paint it. It appears fine in 3d-Coat.



I then export it as a FBX file again with the textures I need. Bring it into photoshop do all the necessary tweaks save as .dds files. Open up blender import the FBX from 3D-Coat and connect the 3 texture files after assigning a pdx material to the object. In this case the “PdxMeshShip” shader. It too seems to be symmetrical at this stage.



I export the file using ross-g’s PDX Blender plugin. I create the .gfx and the .asset files. Nothing is flyable as I have not gotten that far so I just use the “spawnentity” command. But once it is in game it looks wrong on the left side.


In Maya there appears to be a way to “Flip UV Shells” which appears to be missing in blender. Flipping normals on the left side just causes that side of the mesh to not render when in Stellaris, so that isn’t a solution.


Any ideas how to fix my normal map issues?

 

[1codemaster]

Paradox does not support offset UV's like that to my knowledge. If you want the texture mirrored, the easiest way to do so is to unwrap (as you have) then scale -1 on the X axis (S > -1 > Left Click). This will overlap your left and right, but the one you scale will be mirrored. That is the way to "flip" a UV in blender.

 

[StarryWisdom]

I did as you suggested, selected the left side islands, hit S > -1> left click. Needed to then rotate the UV 180 to line up with my textures.

No Dice. Same problem I'm afraid.

The issue I have is the textures, the diffuse and the specular both display properly, but the normal map is inverted on the left side.

So convex becomes concave, and concave becomes convex.

Paradox does seem to allow for offset as I did it this way in part after seeing it used with the Avian science ship.

Avian Science Ship UV (shared by Reiseafa on discord):



But seemingly Maya has a way to flip the "UV Winding Order" as shown in the image above, its what the blue and the red represent.

And I think mine must be the same winding order for both sides....I guess?

I believe that is what is causing the normal map to be inverted. But I have yet to find a solution to this problem in blender.

 

[1codemaster]

Can I take a look at your .blend file? I've never encountered something like that before. Blender does not have UV Winding Order. Im not really sure why you would ever need this option as the "way" to wind counter clockwise is to invert the scale of your UV. This flips your vertices, so they should be wound "the other way," even if they still are wound clockwise. I use symmetric textures, but the way I normally go about it is overlaying the UV's ontop of each other. From the screenshots it should work if paradox supports multiple UV panes ( or whatever you'd like to call them).

 

[StarryWisdom]

Hopefully I did this right, and it has all the relevant files packed with it. I included my mod files for the slipfighter just to be able to see it in stellaris as that is where the normals are misbehaving.

 

[Sid Meier]

For the Blender export/import tool has anyone noticed anything like where for during an animation at random frames there is a flicker/jump like the pose of a bone is out of place for a single frame? Is there a way to "slow down" an animation on the Stellaris side of things?

-Everything looks fine in blender.
-I did the Bake Action + Shift+O steps.
-I don't use IK (FK ALL THE WAY WOOOOOO!!!!)
-As mentioned before I have four pairs of hair meshes; and then 20 other meshes for different body parts (Please no one say 'That's too many!' without knowing the scope and scale of my project unless you know 100% confidence it is related). The issue *only occurs* with the 4th hair mesh pair*. Pairs 1-3 seem perfectly fine with every animation.

I've narrowed down the only as only occurring to a single mesh pair (Hair4) but can't really figure out why it's happening since there doesn't seem to be any rhyme to reason for it.
-It doesn't seem to be a transition issue; it occurs at any random point during an animation;
-it generally seems to be consistently happening at the same frame during a given animation but I can't be sure.
-It only occurs to one particular mesh pair (the front hair mesh and back hair mesh) and none others.
-Fiddling with the transition time seems to change the flickering/popping a bit.

I will fire up OBS later this evening when I return home to record it so I can display the problem and also play with animation playback speed to better see the issue. I might also play with mesh portrait scaling and offset to see if I can get a better look at it.

This issue *used* to appear before with a different mesh setup for hair (Back when I only had 1 mesh pair for all 4 hairstyles and 1 subrig). But deleting it and recreating four different specialized meshes for each hairstyle seems to have fixed the issue for all other hairstyles but one.

I have two main suspects right now:
-Maybe Stellaris doesn't like it when a 2D mesh "overlaps" itself when animated.
-Maybe there is something wrong with my keyframes; or how I set interpolation and Stellaris doesn't like it. I think HairRoot bone in some animations I didn't put a keyframe at the end of the loop, maybe it doesn't like this.

Or maybe, honourable mention, something wrong with the mesh and how it was skinned. It is weird that only ONE particular pair of my hair meshes and no other mesh has a problem and that the problem only occurs for a few random instants during the animations. It doesn't seem to occur with every animation either, only some is it mostly noticable.

Steps to further diagnose I will do:
-Record the issue with OBS.
-Make a "clean" animation with mostly random smooth movements except for Hair4Mesh (leave it unanimated except through its parent bone).
-Make another clean animation with some movements (that dont self overlap).
-Make one with major self-overlaps.

And see if this narrows it down.

Worst case I could leave it in regardless of flickering or take it out/diable it entirely but I am loathed to do so without further investigation.

To repeat I think it's less so much a "flicker" as it is a "pop" like the mesh is moved/reset to the wrong position for a single frame (for a sequence of frames, giving the flickering look) before resuming the animation "normally".

I will investigate and report back, I will appreciate any feedback or help in diagnosing it. I am 99% done though and just have this one last issue to resolve.

 

[ross-g]

Hopefully I did this right, and it has all the relevant files packed with it. I included my mod files for the slipfighter just to be able to see it in stellaris as that is where the normals are misbehaving.

Hey, I checked this out (removed diff and spec for a better look at it) and you are correct. The mirrored side of the ship does look incorrect, with an inverted looking normal map.

(for reference, the avian ship doesn't have the problem)


Offsetting UV's outside of the default (0-1) UV tile is pretty standard, all game engines support this and like you realised it's done on the avian ship.

"UV winding order" and "triangle winding order" just mean whether a face/uv-face triangle is constructed in clockwise or anti-clockwise order. Triangle order is definitely correct in my exporter otherwise all of the face normals would be pointing the wrong way in game compared to in Blender. The red/blue effect you see in Maya is just a visualisation of which way the faces corresponding to your UVs are pointing (either "into" the screen, or "out of" it). If I import your ship using it's .mesh file into Maya then your UV's also have a red half and a blue half, this isn't something that you need to do manually... it just works. For some reason Blender has no preview option for this......


So there are a few possible reasons for your normals looking wrong above:
1) The engine doesn't correctly support mirrored normal maps that were baked from high poly. This could be true ... but doubtful.
2) The exporter doesn't write out tangent data per vertex correctly for mirrored UVs. This is possible, I'm relying on Blender giving me the right data when I ask for tangents... but it might be that I need to manually check for flipped UV's and set the tangent W to -1. I'd need some time to test it.
3) ??? other bugs or normal bake errors.

 

[ross-g]

If I delete a mesh that has an material attached; it should probably automatically remove that mesh from the export list.

Thanks for pointing this out, I did some tests and it was indeed a bug. Apparently deleting a mesh in Blender only removes it from the current Scene immediately... not from the file until you save. So I've fixed this bug now.

For the Blender export/import tool has anyone noticed anything like where for during an animation at random frames there is a flicker/jump like the pose of a bone is out of place for a single frame?

There is a bug where animations can suddenly jump oddly for a few frames yes I've struggled to fix it so far because my maths is correct and I don't know why the data coming out of Blender jumps around... seems a pretty rare bug from what I can tell.
After you have exported everything, try importing your mod files back into Blender using the tool. If it is jumping around weirdly in game then you should also be able to see it jumping around in Blender after you reimport the files.


I still have to say though... you are making waaaay more work for yourself by insisting on doing everything as part of a single skeleton and single model. If you have 4 hairstyles, why not 4 separate rigs & models? Setting some hair meshes to be invisible with alpha is pretty bad as you're still paying the cost for their bones, skinning and verts... having something with alpha on it usually makes it more expensive to render even if you can't see it.

 

[StarryWisdom]

@ross-g - Thanks for taking a look. I was kind of stumped as to why this was happening. I'm kind of new to the whole normal mapping thing. As such I'm a bit out of my depth when it comes to figuring out what might have gone wrong. So I appreciate the explanation.

Add to that, I'm doing all this on an iMac in OS X, so there is a bit more ? when it comes to certain things.

I'm curious, if you import my slipfighter model into Maya and then export it, does it show up correctly in Stellaris?

 

[Sid Meier]

Thanks for pointing this out, I did some tests and it was indeed a bug. Apparently deleting a mesh in Blender only removes it from the current Scene immediately... not from the file until you save. So I've fixed this bug now.



There is a bug where animations can suddenly jump oddly for a few frames yes I've struggled to fix it so far because my maths is correct and I don't know why the data coming out of Blender jumps around... seems a pretty rare bug from what I can tell.
After you have exported everything, try importing your mod files back into Blender using the tool. If it is jumping around weirdly in game then you should also be able to see it jumping around in Blender after you reimport the files.

I did some testing and made new test animations and the issue didn't occur. I think what happened is either:
-Quaternion witchcraft. Sometimes Blender interpolates rotations in a weird way and gotta be fixed manually. This might explain the issue.
-I was very careful to avoid self-overlapping for the two test animations. Perhaps it doesn't like it when a mesh overlaps itself during a keyframe.

What I think I will try is to one by one retouch the animations to avoid overlaps and see if that fixes it.

I still have to say though... you are making waaaay more work for yourself by insisting on doing everything as part of a single skeleton and single model. If you have 4 hairstyles, why not 4 separate rigs & models? Setting some hair meshes to be invisible with alpha is pretty bad as you're still paying the cost for their bones, skinning and verts... having something with alpha on it usually makes it more expensive to render even if you can't see it.

Muh Selectors!!!! Largely this is because if I have four separate rigs and models I can't make use of Selectors AFAIK. This way players can select any of 4 hairstyles, or 12 colour variations or 125 clothes.

Preview (Click to enlarge):


Performance wise I've noticed nothing on my computer, though it is a mid-high end machine with a 9th gen Intel i5 and 32 gb of ram and a nvidia 1080 gtx.

I strongly suspect if there's any performance issues on lower-end machines switching to lower resolution textures would fix it (right now they are all 4096 by 4096!!!!!!)

I think I paid about 300-400$ so far for the art for the mod, so I basically want to go all in as much as possible in terms of QUALITY.

 

[ross-g]

@StarryWisdom

The good news is that I think I have fixed the exporter with regards to correct tangents on mirrored UV's.
The bad news is that my PC has stopped booting which... among other things means I am unable to test it.


I've attached what I think is a fixed export of your ship model. Try it out in game and see what it looks like?

 

[Sid Meier]

I've taken a screenshot at the exact frame in stellaris.



What's weird is there is like a 30-40% chance a given loop of the animation seems to play normally without issue. It's quite bizarre as it doesn't happen *every* time just *most* of the time. Like Stellaris only renders from number of frames so there's a chance that the given range of frames that have the issue don't get renders and things look fine.

Based on playing with the bones and looking at the F-Curve window; maybe the issue is that a tiny rotation along the wrong axis leaked out and Stellaris or the Mesh converter is freaking out about it during those frames? It's like a tiny epsilon difference from 0.00 to -0.00 along Z. I'm around 90% sure it's something along these lines; something that's not visible in the Blender playback but shows up in game engines.

 

[StarryWisdom]

@ross-g

You fixed it!!!




Its perfect now.

Thank you, so much!

Sorry to hear about your pc. Hopefully its not something too serious or costly.

 

[Sid Meier]

So far going into the f-curve menu and playing around with I think "Smooth Keys" seems to work... But in a very destructive way. But it does seem like if I just carefully redo my animations that seems to fix it.

It's like the gimbal lock problem where the quaternion for a single frame seems to interpolate the completely wrong direction or is very similar to this problem but I have yet to solve it in a non-destructive or non-tedious means.

Test your animations one at a time folks I guess.

Hrm, one thing I should test if just deleting the offending keyframe and re-inserting it fixes it.

Edit2: Seems like switching to Linear interpolation seems to fix the issue with only minimum destruction to my overall animation compared to Smoothing Keyframes.

 

[1codemaster]

i think i have diagnosed the issue somewhat. further testing reveals i can't reproduce the bug if the full rotation is asymmetric. So when you go from rot 1 > 2 > 1 again it breaks sometimes, but if you go from 1 > 2 > 3 it never breaks
and it breaks when the animation curve is bottomed out at dR 0

Quaternions (which you should be setting your keyframes as) do not gimbal lock, they are completely smooth transformations because they aren't three dimensional, it's more of a twist that extends "forwards" in space. Blender uses unit quaternions, so this twist just loops back on itself.

This is the only circumstance I can reproduce the bug in, but not 100%. It only happens sometimes for some bones. From what you said Sid it seems like the issue is the animation curve bottoming out (no rotational acceleration), as linear interpolation removes these areas. The easiest fix I have found for right now is to add another intermediate keyframe slightly offset from the others. For example to sway left to right, having a keyframe in the center slightly forward will almost always eliminate the issue, so you mesh moves in a triangular pattern, opposed to a purely linear one. Another thing to look at would be bone structure. I've only see this bug happen on legs mostly, and an "arm" once (not really an arm). Maybe it is a result of the way joints are different in between applications, as there is no head/tail, only 'joint.' It could also be the looping nature of Blender's unit quaternions, very difficult bug to catch, let alone fix.

I hope you can get your computer fixed ross that sucks!

 

[Sid Meier]

Thanks Code for looking into the issue!

Yes, for all of my pose rotations they are all Quaternions. I know Quaternions are meant to *solve* gimbal lock, I'm just explaining the observations I'm making where for those specific frames where the rotation just spazzes out.

So far I'm trying to add keyframes move them around but so far no luck.

An additional annoyance is apparently selecting *every* keyframe and setting to linear doesn't work, but I have to correctly ascertain which keyframe is the culprit and only make that one linear.

Example (Click to enlarge):


I marked the keyframes I added with red dots on the right. I moved them slightly to the right or left 1-2 frames.

I marked on the left which bones were the offenders. The hair on the "right" or my character's "left".

Hair4_2.L f-curve


Bangs.L f-curve:


If I click on normalize I get this strange valley, but the actual issue approximately occurs in the area marked a little to the right at frame 287:



And yes to be clear the issue is literally only occuring for the ".L" bones, right side is fine (for this animation; but also generally only for these bones for this hair style).

Edit to add: Regarding my skeleton, I moved my bones to armature bone layers to make things easier to manage, but basically the "root" bone for each strand here, so Hair4_0.L, Hair4_0.R, Bangs.L and Bangs.R, and Brow, are all parented to a currently hidden "Hair_Root" which is parented to the head.

So here we have a chain of three bones for both strands like a typical arm setup.

 

[Sid Meier]

Another question but is the "Bake Action" + "Sample Frames" steps actually needed using Ross's plugin? As a test I clicked on export animation and aside from my currently ever-present issue that's present regardless it seems to export just fine?

 

[Sid Meier]

Oh I think I managed to figure out how to apply Code's solution.

I think it wasn't enough to just insert a keyframe and then "shift it" left/right on the dopesheet. But to actually grab the point on the F-Curve (say the X quaternion component) and then move it up/down a couple of sticks to create the more "triangular" looking curve.

AFAIK this should do weird things to the quaternion but it seems to work?

 

[mralexs]

Question, how does one assign multiple diffuse maps to a model for this?