This time I succeeded with two quite big things in my workflow. Finally I managed to bake the height map without any weird gradients across it and I found a better way to remove ao shadows from albedo then using 'ao cancellation' node.
Still need to process a few more tests but looks like I made a big step forward :).
The secret ingredient to get nice and clean heightmap for flat surfaces is simply to decimate existing highpoly model into a lowpoly one.
I used RIZOMUV to uvmap it later with one click :)
Also I improved the way to remove ao shadows simply by using AO as a mask for 'color equalized' albedo :) and the result looks great so far. The level of details can be modified by 'Equalise' node same as amount of blending in 'blend' node.
Hope this is going to help anyone to make even better materials :)
If anyone is interested can record another boring tutorial to cover this process in details :), just let me know.

his is my photogrammetry workflow - still experimenting with 'Artomatix' and replaced Photoscan with Metashape (more details regarding to my workflow at: https://www.artstation.com/artwork/K0PYG)