The Test Setup
I used two C track circuit tracks with a 24188 between. I ran various locos across this test track at speed step 1 (exceptions are indicated) of the best protocol support by the decoder.
The freeware PaTacho was used to measure the time required for each test run.
PaTacho automatically calculates the scale speed (prototype speed in km/h) and the values were recorded manually.
Each loco ran 8 times forward (left to right) and 8 times in reverse (right to left).
Some motors are not really symmetric, leading to higher speed in one direction. So for each loco I look at the result for forward only, for reverse only and also for all test runs. If loco is faster in one direction this is not held against the decoder.
The locos ran single without load. The same test track and the same configuration was used for all test runs.

This test does not indicate whether a loco runs smooth or jerky or whether the transition from standstill to running is smooth or jerky.
This test does indicate how strict a decoder can control the speed of a loco.

Highly interesting for me are results of otherwise identical locos with different decoders.

Expectations
Expectations are that an excellent decoder will manage all test runs within +/- 5% of the average speed. Up to +/- 10% will be allowed for good decoders.
Variations larger than 10% are considered lack of load regulation.

Results

The test roster included three Märklin Ludmillas: One with factory factory-installed fx decoder, one with factory-installed mfx decoder (sound decoder), one with user-installed ESU V4 LokSound decoder. As all three locos share the same cheap three-pole can motor, I was very interested in the results of the comparison of these three locos.
The fx decoder does not allow load regulation parameters to be configured. The mfx decoder was used with the factory-default settings. The ESU LokSound decoder was used with load regulation parameters I set after a few test runs (these may not be the best parameters though).

Ludmilla fx (factory-installed)
Forward
2,33 2,28 2,24 2,21 2,21 2,21 2,2 2,21
Average 2,23625 Min 2,2 Max 2,33 Max/Min 5,91% Max/Avg 4,19% Min/Avg -1,62% STDABW 0,045961941
Reverse
2,4 2,38 2,35 2,34 2,32 2,31 2,3 2,31
Average 2,33875 Min 2,3 Max 2,4 Max/Min 4,35% Max/Avg 2,62% Min/Avg -1,66% STDABW 0,036030741
Total
Average 2,2875 Min 2,2 Max 2,4 Max/Min 9,09% Max/Avg 4,92% Min/Avg -3,83% STDABW 0,066282225


Ludmilla mfx (factory-installed sound decoder)
Forward
2,36 2,47 2,83 2,45 3,21 2,52 2,69 2,25
Average 2,5975 Min 2,25 Max 3,21 Max/Min 42,67% Max/Avg 23,58% Min/Avg -13,38% STDABW 0,30681544
Reverse
2,85 3,01 3,27 3,46 3 3,09 3,04 3,86
Average 3,1975 Min 2,85 Max 3,86 Max/Min 35,44% Max/Avg 20,72% Min/Avg -10,87% STDABW 0,326091573
Total
Average 2,8975 Min 2,25 Max 3,86 Max/Min 71,56% Max/Avg 33,22% Min/Avg -22,35% STDABW 0,435377231


Ludmilla ESU V4 (user-installed)
Forward
1,55 1,56 1,56 1,57 1,57 1,57 1,57 1,57
Average 1,565 Min 1,55 Max 1,57 Max/Min 1,29% Max/Avg 0,32% Min/Avg -0,96% STDABW 0,007559289
Reverse
1,56 1,56 1,57 1,57 1,57 1,57 1,56 1,57
Average 1,56625 Min 1,56 Max 1,57 Max/Min 0,64% Max/Avg 0,24% Min/Avg -0,40% STDABW 0,005175492
Total
Average 1,565625 Min 1,55 Max 1,57 Max/Min 1,29% Max/Avg 0,28% Min/Avg -1,00% STDABW 0,006291529



Another interesting test field: three Köf II and one Ka. I presume they all have the same motor.
I have to assume that two Köf II and the Ka have been made in China while the latest Köf II probably was made in Hungary. Looking at the motor will void the warranty, so I can only look at the test results.

Tm 34 SOB (60901 with DIP switches, Speed step 2/27, loco did not move at speed step 1)
Forward
4,19 4,24 4,21 3,97 4,17 4,18 4,16 4,06
Average 4,1475 Min 3,97 Max 4,24 Max/Min 6,80% Max/Avg 2,23% Min/Avg -4,28% STDABW 0,088761317
Reverse
3,82 3,79 3,75 3,74 3,74 3,74 3,74 3,74
Average 3,7575 Min 3,74 Max 3,82 Max/Min 2,14% Max/Avg 1,66% Min/Avg -0,47% STDABW 0,030589447
Total
Average 3,9525 Min 3,74 Max 4,24 Max/Min 13,37% Max/Avg 7,27% Min/Avg -5,38% STDABW 0,211360671

Köf II Wiebe (PIC)
Forward
2,37 2,37 2,37 2,37 2,38 2,37 2,38 2,37
Average 2,3725 Min 2,37 Max 2,38 Max/Min 0,42% Max/Avg 0,32% Min/Avg -0,11% STDABW 0,0046291
Reverse
2,48 2,48 2,49 2,49 2,47 2,45 2,47 2,47
Average 2,475 Min 2,45 Max 2,49 Max/Min 1,63% Max/Avg 0,61% Min/Avg -1,01% STDABW 0,013093073
Total
Average 2,42375 Min 2,37 Max 2,49 Max/Min 5,06% Max/Avg 2,73% Min/Avg -2,22% STDABW 0,053774219

Ka mfx (ESU)
Forward
0,84 0,85 0,85 0,85 0,83 0,85 0,85 0,85
Average 0,84625 Min 0,83 Max 0,85 Max/Min 2,41% Max/Avg 0,44% Min/Avg -1,92% STDABW 0,007440238
Reverse
0,83 0,83 0,84 0,84 0,83 0,84 0,84 0,84
Average 0,83625 Min 0,83 Max 0,84 Max/Min 1,20% Max/Avg 0,45% Min/Avg -0,75% STDABW 0,005175492
Total
Average 0,84125 Min 0,83 Max 0,85 Max/Min 2,41% Max/Avg 1,04% Min/Avg -1,34% STDABW 0,008062258

Köf II mfx (Märklin)
Forward
3,34 3,34 3,33 3,33 3,32 3,26 3,34 3,35
Average 3,32625 Min 3,26 Max 3,35 Max/Min 2,76% Max/Avg 0,71% Min/Avg -1,99% STDABW 0,028252686
Reverse
3,39 3,4 3,4 3,44 3,44 3,43 3,43 3,4
Average 3,41625 Min 3,39 Max 3,44 Max/Min 1,47% Max/Avg 0,70% Min/Avg -0,77% STDABW 0,020658793
Total
Average 3,37125 Min 3,26 Max 3,44 Max/Min 5,52% Max/Avg 2,04% Min/Avg -3,30% STDABW 0,052265349


Here are the results for the new Lollo Märklin 37740, Loco of the month - and for me the biggest disappointment this year so far.
Lollo 37740
Forward
4,41 3,83 2,98 4,3 2,96 4,53 3,06 3,15
Average 3,6525 Min 2,96 Max 4,53 Max/Min 53,04% Max/Avg 24,02% Min/Avg -18,96% STDABW 0,689715262
Reverse
3,38 2,76 4,43 4,26 3,39 4,42 4,45 4,43
Average 3,94 Min 2,76 Max 4,45 Max/Min 61,23% Max/Avg 12,94% Min/Avg -29,95% STDABW 0,663497228
Total
Average 3,79625 Min 2,76 Max 4,53 Max/Min 64,13% Max/Avg 19,33% Min/Avg -27,30% STDABW 0,670431453


Conclusions
The comparison of the three Ludmillas (different decoders, but probably same type of motors) confirms what I expected from eyesight: the fx decoder controls the speed of the loco pretty well (within +4.92%/-3.83% of the average). The ESU V4 beats it slightly (+0.28%/-1.00%).
The bronze medal goes to the Märklin mfx decoder (excellent load regulation, all test samples were within +33.22%/-22.35% of the average). Not even two orders of magnitude worse than the winner of the gold medal ... (do I sound sarcastic?)

The comparison of the four Köf II/Ka (different decoders, but probably same type of motors) leads to slightly different results: the new Märklin mfx decoder controls the speed almost as good as the ESU mfx decoder (but is about four times as fast at speed step 1 out of 126). The programmable fx decoder (PIC decoder) is also slower than the new Märklin mfx decoder and achieves slightly better results (reverse speed is higher than forward speed, but I don't blame the decoder). Results for the 60901 decoder vary a bit more. But the worst decoder out of these four still shows good results (+2.23%/-4.28% forward, +1.66%/-0.47% reverse).

The Lollo Märklin 37740 stands alone in this test field. I'm so disappointed about this loco that I started this test series to get figures that show her weaknesses. The transition from standstill to speed step 1 is very jerky with this loco (which you cannot tell from the test results shown here) and the speed at speed step 1 varies a lot. She runs smoothly at a speed that appears to be constant - but each time I start her she runs at a different speed.
The eight forward samples fall within +24.02%/-18.96% of the average, the eight reverse samples fall within +12.94%/-29.95% of the average. For the total of 16 test runs, this gives +19.33%/-27.30%.
The fastest forward test run was 53.04% faster than the slowest run, in reverse it was 61.23%.
This is not what I call excellent load regulation. She runs pretty fast at speed step 1 - the average is close to 4 km/h.


Today I tested five BR 101 models from Märklin. All five should have the same motor, but came with different factory-installed decoders: 60901 with DIP switches, fx PIC (programmable decoder), ESU mfx, MäTrix mfx with and without sound.
I also tested one BR 120.0 with factory-installed ESU OEM decoder and another BR 120.0 which has an ESU LoPi v3 instead.

Here are the results after 23 test series with 21 locomotives:
Locomotive (Decoder), Variation of speed, Scale speed (km/h)
Lollo 37740 (DCC ESU LoPi 4 (default values)) 0,31%, 3,22
BR 101 37397 (fx PIC) 0,34%, 5,98
BR 101 37399 (mfx ESU) 0,41%, 3,63
Ludmilla 36423 (DCC ESU LoPi V4) 0,48%, 1,57
BR 101 60901 37391 FS2 (fx DIP (speed step 2/27)) 0,48%, 10,36
Köf II Wiebe 36820 (fx PIC) 0,51%, 2,42
120 004 37537 LoPi 3 (ESU LoPi v3) 0,58%, 3,49
BR 101, 37370 (mfx Sound Trix) 0,82%, 3,59
Ka 36810 (mfx ESU) 0,89%, 0,84
Köf II 36827 (mfx Trix) 1,04%, 3,37
E 17 37061 (mfx Trix) 1,35%, 2,90
BR 216 mfx 37748 (mfx OEM ESU) 1,44%, 2,96
BR 216 PIC 29710 (fx DIP) 1,83%, 5,74
120 002 37538 original (fx OEM ESU) 1,94%, 4,10
Tm 34 SOB 36803 (Speed step 2/27) (fx DIP) 2,16%, 3,95
BR 101 37390 (mfx Trix) 2,34%, 2,95
Ludmilla fx 36420 (fx PIC) 2,52%, 2,29
Trix ES 64 U2 with MLD (mfx Trix 60942) 4,99%, 3,06
BR 101 60901 37391 FS1 (fx DIP) 7,28%, 1,05
BR 216 DIP 37744 (fx DIP) 7,87%, 1,24
Ludmilla mfx 36424 (mfx Sound Trix) 17,14%, 2,90
BR 103 37573 (mfx Sound Trix) 17,28%, 2,93
Lollo 37740 (mfx Sound Trix) 21,47%, 3,80

Remarks:
The best results were achieved with the Lollo 37740 with a temporarily installed ESU V4 decoder; the worst results were achieved by the same loco, but with the factory-installed decoder.
The new mfx decoders are listed as Trix decoders because they return the manufacturer ID 131 which is Trix. The slow speed control of these decoders varies a lot: the best result was achieved in a BR 101 with DCM (0.82% variation), the worst result was the Lollo with 21.47% variation. All results greater 10% came from the new mfx sound decoders (and smaller values are better).
I tested the BR 101 37391 twice: once at speed step 1 where it ran very slow (around 1 km/h), but not smooth and once at speed step 2 where it ran very smooth, but rather fast (around 10 km/h).
You cannot tell the smoothness from the table above. Personally I'm very impressed by the performance of the Ludmilla with the ESU V4 decoder: it ran smooth at a constant and rather slow speed.
The Köf II with the Trix mfx decoder was nearly as constant as the Ka with the ESU mfx decoder - but the Köf II ran four times as fast (and constant running at slower speeds is more impressive IMHO).

Edited by user 26 December 2013 21:11:10(UTC)  | Reason: More results.

I still want to do some more test runs with the 6090 DCM. Maybe my class 101 locos will be guinea pigs for that test.

Now there are results for 8 locos in the first post.

Hi Tom,

Thanks for your methodical testing.

I've just been running my 37441 Lollo, and by watching very carefully I noticed a slight unevenness in the running at speed step one, which could have been due to the long train it was pulling. I did not notice any inconsistency in the minimum speed when starting and stopping a few times, and the mimimum speed always seemed to me to be about walking pace. This is higher than, for example, the Br221 3782 I have on the track at the same time, but is still acceptable in my eyes. The Br221 with a 60902 decoder and 5 pole DCM motor was smooth but made groaning noises at speed step one (probably needs a drop of oil).

My conclusion is that your test results show a worse performance for this type of loco which I can partly confirm, but not to the degree you observe in yours. Either you have an unusually bad example or I have a particularly good one. Either way, it shows that somewhere in the production of this loco there is a variation in tolerances that needs to be addressed. Whether it is in the motor, the gears, or the decoder is not obvious to me.

I've always had a suspicion that the 6090/60902 decoders were the smoothest made by Marklin (including ESU imports). Perhaps you can extend your tests to check this out?

As far as possible I used Märklin locos with factory-installed decoders and always with factory-installed motor.
I test the locos with the default settings from the factory. There are reports that some of the new locos run better with modified decoder settings - that might be something for further tests in the future.

For the time being I expect Märklin to assure that the decoder settings are optimal and that the motor and gear are flawless.

Maybe I'll send my test results to Märklin and ask them what they think about it (I've had bad runners come back from Göppingen unmodified with the remark "matches the series standard" and so far I fear they'll tell me the same about my Lollo).

this kind of test would have been more convincing if you tried 3 decoders on a single locomotive model.... and if you did it with another central like the Marklin CS2 too. (test with 2 centrales)
this test shows the behavior at speed step 1 . Not more.
Please could you give the references of the 3 models : 36xxx ?

....I quote myself to repeat my question in a different way
There were around ten or a dozen of Marklin Ludmilla , which Ludmilla model have you been using for each decoder, please ?

Hi all,
Tom,I think tests like that will really help the nuts,er,folks here! Thank you!
Funny,I just finished installing a TCS M-1( http://www.tcsdcc.com/public_ht...le/M-Series/M-Series.htm) in a FL 4155,no CV changes and smooth as silk.Thanks again for your efforts!

Dirt (yes it weathered,..... )

I used to be agravated that I had to replace the MFX decoders with ESU V4s so I could run my lokos with DCC. Your tests, the greater adjustment range in sound file volume and the ability to easily create sound projects makes me much less agravated.

good work!

The first thing I did back in 1989 was to graph the speed steps of the 6080 decoder.
I also found forward and reverse differed so I had to add reverse speed step calibration to my software.

I also found that as the loco warms up its speed increases.

http://layout.mixmox.com/1/ICE_speed

Hi Tom,

Thanks for doing these types of test

I have long awaited some benchmark testing of the products we buy so we can compare and make an informed decision about our purchases.

I am by no means an expert but have done some speed profiling with PC software and have few concern regarding the accuracy of the results.

As I understand the actual track length you use for measuring is about 29 cm and the loco’s are running at speed step 1?

Generally you would get more accurate results with a longer distance between the 2 triggers (circuit track in your case). You are comparing speed over a short distance and the inconsistence in when the software actual register the puls from the circuit track will skew the results compared to doing it over a much longer distance.

Also I am not sure if running a loco at speed step 1 has much to do with load regulation? I would imagine this has more to do with the actual performance of the engine and its gearing. At speed step 1 the engine is rotating very slowly and any slight obstruction in gearing etc. will influence the result (random reults).

As I understand load regulation has more to do with consistent speed when load is increased or decreased of the loco (e.g. when climbing up a hill and running down again)? The decoder sends a command to engine – then listen – if the engine is not doing as told it will send a signal to correct it? When running at speed step 1 we are not really exposing the loco to any change in load?

I would suggest a longer track length with e.g. a climb and running the engine at medium speed, where the engine will be exposed to an actual change in load. I believe this will give a more accurate comparison of load regulation performance, if that is want we want to compare?

Brgds - Lasse

I started these test series because I noticed that the new mfx-equipped locos sometimes start with an abominable jerk (sometimes with a small jerk, sometimes jerkless) and then continue to travel at varying speeds.

The jerk is just a short moment of inconsistent acceleration. I found no easy way to measure that yet.
But I can easily measure the average speed of the loco across my test track.

At speed step 1, we are talking about times of 25 through 100 seconds! I use an old MS-DOS PC that does nothing else but poll the circuit tracks a few thousand times a second. I've been using this test configuration for years now.
In the beginning I even connected the circuit tracks to two PCs at the same time. For short intervals (fast locos), the difference normally was 0 to 2 ms.
A PC clock can be expected to be off by 30 seconds a day, so you have to allow 1 ms difference for each 3 seconds of measured times. And the results of the two PCs were only few milliseconds apart, as expected.
Good locos will give results that vary very little. I have no doubt about my measuring equipment.

As said before, running at speed step 1 is the biggest challenge for load regulation. I tried to make it simple (no grade, no curves, no load).

I made the tests because I noticed with my bare naked eyes that the speed at step 1 was not consistent. And maximum speed 60 % above minimum speed can be noticed.

The real problem: unsmooth start from stop to speed step 1 => I cannot measure this easily. Tried something with a video camera, but evaluating the videos takes a lot of time.
Another observation: locos that sometimes start jerky also have varying speeds when cruising constantly at speed step 1 (yesterday I also made a testrun at speed step 2 of 27 using MM protocol and the variation was still rather huge; the minimum speed at speed step 2 of 27 was lower than the maximum speed at speed step 1 of 27).

I takes a good motor or a good load regulation to start a train from a stop without jerk. Märklin now sell locos with cheap motors and homemade mfx decoders that lead to locos that start jerky.
The hobby locos (BR 185, 285, ER 20, Ludmilla) run fine with fx decoders, but not with mfx decoders. Same problem with e.g. BR 216/218 (with the small difference that these locos have RRPs of 250+ Euros).

Locos that start jerky are intensively discussed on Stummi's Forum. This finally led to the initiative to send Märklin an open letter about their motor and decoder problems.
It seems only few users here notice the jerks when locos start or care about the jerks. But more users care about cheap motors in high-price premium models.

A loco that starts smoothly (without jerk) at 1 km/h scale speed looks much better to me than a loco that starts with a jerk at a minimum speed of around 7 km/h.

Only one company sells locos that have these abominable jerks out of the box - and only since they introduced cheap decoders in 2009 and combine them with cheap motors since around 2011.
I know now how to avoid these jerks.

Or time for Märklin to improve the decoders - or use better motors while the decoders are not on the level.

Not all customers have problems with the new locos, so maybe it's profitable for them even if they lose a small minority of critical customers ...

I am impressed your test results. Specially the different behavior of decoders dependent upon the controller station.
Märklin needs to go to a higher standard in decoders, for sure.
Motors are not so critical, but better motors assure better results. I still prefer the SDS and I don't understand why some 2 rail people don't like it.

About controller stations I am a bit surprised between the differences of CS2 and MS2. A little out of topic I also never accepted the light flickering of 6090 decoders in the Central and Mobile Stations. I know it can be solved getting the outputs in other points of the PCB, but that is not acceptable.

I think it was better to standardize a benchmark like yours, but preferably more elaborated with ramps, to which decoders conform (or not).
Also controllers should have something similar, but that is maybe not so easy because DCC is already a standard and MFX is proprietary.

Once more, thanks Tom for your tests and also for let us know the results.

Hi Tom,
Any chance the flickering and maybe some of the "jerking" are related to the power supply? Is the CS2 using a switched mode supply and the others using conventional transformers?

Best regards
Roger

don't forget one thing : at low speed 3 pole motors tend to stall easily when voltage drops (or with voltage cuts/micro cuts),... more easily than 5 pôles motors.
The main shortcomings of 3 poles motors are felt at low speed, principally.

Hi!
This quote is from a different thread, but I think I should answer it here:
The inconsistent speed at speed step 1 is a minor issue for me. It would not prevent me from buying a loco.

But locos that fail to start moving smoothly without a jerk are a significant issue for me. And so far I assume that both problems have the same origin: a sloppy, faulty load regulation that prevents locos from starting without a jerk (not even the jerk is consistent, but most of the time there is a jerk) and also prevents them from running at a consistent speed at speed step 1.

It's a big issue for me that my former MRR main supplier now favours profit over quality in an extent that leads to poorly running locos.
Locos still run better than Delta locos or Piko locos with AnDi decoders. But Märklin locos with an RRP of € 280 run worse than Piko locos with an RRP of € 120.
Now guess where my 2014 pre-orders go ...

A new feature coming to the CS2 soon: a "macro recorder": you run a loco manually, the CS2 records what you are doing and can replay it later. Great feature - and obviously it works better if locos have consistent speed.

Yes, the problem is with the mfx decoders only. I hope a firmware upgrade will solve the issue. For decoders sold separately, the firmware upgrade can be done with the CS2 (60940, 60942, 60948, 60949 etc.).
Locos with pre-installed mfx decoders however will have to go back to Göppingen - or at least their decoders.

No problems with my mfx-free TRAXX and Ludmilla locos. Only TRAXX and Ludmilla locos with factory-installed mfx decoders or converted with 60948/9 run too bad for my taste.