I've not yet tried my hand at it, but like you , have seen a lot of videos of other people's rendering of mountains, and of general stone faces (exposed hillsides and rock cuts etc). Some are great, others not so much, but in the end the only view that matters is the owners view. From looking at the many videos some of the thoughts that crystalized in my mind that I'll be using when I get to that stage (as there are helixes and tight bends to hide) are:
1) Decide what type of stone to model (limestone is different than granite is different than etc) and look at pictures of what that type of mountain stone (and soil and trees/plants) looks like.
2) Look at the foliage and coverage by soil and plants as well as the rock/mineral shapes - you find most mountains have a lot of plant growth on them unless you are modeling only above the snow line.
3) Pick a few focus areas where the exposed stone will be clearly visible - work carefully on that part - other parts likely will be under soil/grasses/trees and so the exact shape is less important once you apply the landscaping.
4) Practice - build some small diaramas, repeat, get better, repeat - you can do this at a kitchen table in comfort without leaning over and stressing about the layout. Like golf, no amount of watching videos can make up for doing it yourself (so just keep trying)
5) Think about the proximity of the viewer - often times what doesn't look very good up close, from 10 ft back looks great, and so modeling the forest parts or other aspects of a mountain, when viewed from what would be many miles away from the viewers perspective, isn't in need of so much detail; you can get away with much simpler techniques and materials in the background.
6) Don't necessarily model the whole mountain - they are huge, and you, in the railroad scale, are realistically only going to build a small face or tiny fraction of a mountain, not the whole thing from valley floor to peak. That gives you much more room to work in terms of what you need to actually depict over a square meter area.

Another thing I just remembered there are now several types of material that work well with different techniques:
1) Traditional cardboard or screen wire mesh base with plaster cloth or plaster (in layers) applied, where one, when applying thicker (inches thick) can as it is firm but not hard (15mins?) - chisel cuts into it.
2) Styrofoam - cheap as it comes with so many products, but messy as hell; cover it with wood putty to smooth or plaster after general shaping
3) Extruded foam insulation sheets - build it up and can cut and carve with less mess
4) Expanding construction foam in a can - spray it down to a rough shape (it expands many times its initial size) and then carve with box cutters, blades, rasps, knives (you get opened air pockets, so you may need to add a skim coat of plaster or wall board joint compound if you need smoother surface).
5) paper/fiber based wall/sound boarding can work well as layers of limestone and similar layered rock in hilly areas with rock cuts.

I use the method of Bernhard Stein. (-same one used by brima and MiWuLa);

A fine-masked metal net, where the main forming can be done in 3D.
Then covered with paper (newspaper soaked in glue, or other things to close the masks of the net. (I think brima doesnt use anything other than a layer of thick plaster)

Then cover everything with plaster. Especially thick where you want to make many details.
Once the plaster is dry (or almost dry), you use a chisel to carve in the structure of the rocks.

Paint it all gray/brown.
Wash it with black.
Dry-brush with white.
Add vegetacion on (almost) horizontal parts.

This gives quite a good result, even if you are not very experienced in modelling. (Once you have experience it gives excellent results ;-) )
Drawbacks are that it weighs a lot, and you need to clean up the left-over plaster.

This method is very well documented in Youtube.

Great suggestions above. Another consideration is do you want permanent mountains or mountains you can adjust? If you make your mountain out of foam that can be lifted off the layout, then you can remove the mountain, and take it out to the garage to detail it. Or if you don't like the way it turns out, you can make a new removable mountain. Removing a permanent mountain that isn't to your liking is a messy job.

Here is a video (German) of a guy using spray foam to lay the basic contours of mountain surfaces on wire mesh, then shaping that then covering it with a thin layer of plaster. The motivation behind using blocks of foam or spray foam is that using foam filler gives you solid support behind the plaster, and very light weight vs the thicker plaster builds.

Thanks Dale,

I painstakingly did my best to copy the actual contours of the hill/mountain the castle sits on. It's amazing what you can see with the 3d imaging on google maps. Due to constraints of size the hill isn't 100% accurate, but a very good representation.

Hi 57vert.
Search the forum for my posting by YannisB “Building my new layout”. Scroll to pages 3 and 4 of the posting and you will find my photos and explanations how I made my mountains.
Message me and will be happy to discuss more details.

Yannisb