what do you mean by stone dust, if you are talking about crush sand which contains particle from size 4.75mm to 75 micron, You can use crush sand for doing brick work.
Logical reason - bricks which are used (clay bricks) has genrally strength of 3.5 N/mm2.
Plaster done on wall do have strength of 3.5 to 4 N/mm2 at 28 days.
Using a strong mortar than bricks leads to cracking when there is shrinkage.
A mortar can easily achieve the strength of 4 N/mm2 at 28 days.
So using a mortar of cement and crush sand at proportion of 1:5 or 1:6 (base on fine content in your crush sand) will not cause any issue in it.