- Reference Benchmarks :.
Cross Platform DAW Performance - Part V :
AVID's release of Protools 9 has been one of the most anticipated and surprising moves in recent audio history , breaking away from the past model where the software was always locked to its own proprietary hardware and allowing the use of any 3rd party ASIO/Core Audio hardware has literally changed the DAW landscape overnight. Now I'm not saying they have done anything revolutionary, quite the contrary, for many they are still playing catch up in many areas , ADC/PDC on the Native platform being one example. However, unbolting the software and hardware has opened up opportunities for many end users where Protools was a secondary but necessary tool in the kit, to use their preferred audio hardware instead of what many considered an oversized copy protection device in the form of an interface that they only used to run the software.
That alone has caused a huge shift , not to mention
the inclusion of ADC and most of the other toolkits as standard ,
shows that AVID have managed to turn the big ship around , are listening
to their end user base, and are really trying to reinvent them selves
after the previous M.O was proving to be on the nose for many. This
is the first time I have been able to test Protools on a really level
playing field in regards to preferred hardware , the M-Audio DAE
testing showed some promise, so it was time to roll out ASIO/CoreAudio.
|Preparing for Battle :.|
There really wasn't much at all to do to prepare for this round of testing as apart from Protools 9 , all of the results and testing has been carried over from Round 3.
In the last round I did shift to Cubase 5.5 and StudioOne
1.6 , but in regards to Cubase/Nuendo, the audio engine /optimization
/ performance is near enough to identical , they are basically the
same application after all. In that respect, the earlier N5 results
were fine to recycle
All that was needed was to ensure that Protools was working as designed with the RME HDSPe under both ASIO and Core Audio.
After configuring the application and running some preliminary tests across both Win7 x64 and OSX to ensure no obvious surprises needed to be navigated, I simply knuckled down and for the first time was able to run Protools with the reference hardware and driver protocols used for all of the other DAW's in the test pool.
This is a truly level playing field , and something that for the longest time, many were never expecting to happen.
Lets see if AVID have managed to pull a rabbit out and go toe to toe with the other DAW's on their home ground.
|Round 1 : DAWbench DSP : WaveArts MD5 Multiband Compressor : RME HDSPe AIO :|
With Protools 9 being on the same playing field as the other DAW's now running under ASIO/Core Audio, lets take a closer look at the comparative cross platform results for all the respective DAW's.
Protools 9.0 : @032 no comparative was available as that setting was unavailable on OSX, @064 performed better on Windows by 652%, @ 128 performance was better on OSX by 1 plugin ( .06%) , safer to say on par, @ 256 performed better on OSX by 15%
Nuendo 5 : @ 032 - performed better on Win7 by 240% , @ 064 - performed better on Win7 by 57% , @128 - performed better on Win7 by 21%, @ 256 - performed better on Win7 by 6 %
StudioOne 1.5 : @ 032 - performed better on Win7 by 89% , @ 064 - performed better on Win7 by 68% , @128 - performed better on Win7 by 57%, @ 256 - performed better on Win7 by 45 %
Reaper 3.6 : @ 032 - performed better on Win7 by 47% , @ 064 - performed
better on Win7 by 16% , @128 - performed better on Win7 by 20%,
On Windows the performance for PT9 under ASIO was not as consistent as with DAE. The performance at 032 was definitely an eye opener, showing all the other app's a clean set of heels by 15%, but the scaling dropped off above 064 quite substantially to the point that it dipped below Reaper and was rounded up by Nuendo and StudioOne by 256. Still a pretty impressive start for a first time run on 3rd party ASIO hardware
On OSX , the lowest buffer setting available was 064, so we couldn't have a head to head comparative @32 unfortunately, and the weird anomaly we witnessed at 064 with the DAW engine was also present again under ASIO, very strange to say the least.
Once we got past that @128 and @ 256 the Protools9 / RTAS again easily out performed the rest of the field quite substantially. We do need to remember the looping glitch that I reported on in Part 3 that has held back the potential results for Reaper
So in summary, the results show a more consistent result on Windows for the Protools 9 despite the foot coming off the pedal above 064 , on OSX, it was a mixed bag again - a huge dip @ 064 sample buffers, and then a clear set of heels @128/256 to the point that @ 256 it was 15% above the Windows result.This was consistent to the DAE result from the previous round.
The cross platform performance is again far more even in comparison to the other 3rd party DAW's , so the variable of the DAE engine over ASIO is looking less likely as being one of the possible variables for the more consistent cross platform result , which then leaves RTAS and the actual application coding/optimization as the possible contenders.
Lets move onto the next round and see whether any emerging patterns are maintained.
|Round 3 : DAWbench DSP : URS Channel Strip Pro : RME HDSPe AIO :|
Lets take a look at the respective results.
Protools 9.0 : @032 no comparative was available as that setting was unavailable on OSX, @064 performed better on Windows by 29%, @ 128 performance was better on Windows by 6% @ 256 performed better on Windows by 2%
Nuendo 5.0 : @ 032 - performed better on Win7 by 201%
, @ 064 - performed better on Win7 by 84% , @128 - performed better
on Win7 by 48%,
StudioOne 1.5 : @ 032 - performed better on Win7 by 103% , @ 064 - performed better on Win7 by 104% , @128 - performed better on Win7 by 85%, @ 256 - performed better on Win7 by 81 %
Reaper 3.6 : @ 032 - performed better on Win7 by 40%,
@ 064 - performed better on Win7 by 47%, @128 - performed better on
Win7 by 47%,
On OSX, the results for Protool 9 were very similar to the performance of the DAE M-Audio results, being a lot more consistent across the board and easily outperforming all of the other applications @ the available buffer settings by quite a substantial margin.
The Windows results were the ones that were the mixed bag this time around. Protools 9 again showed all of the other app's a clean set of heels @ 032/064 and was looking like it was going to clean the slate if it continued to scale accordingly , but then @128 the scaling took a bizarre U turn.
My test procedure is to set the buffer to the lowest available, then incrementally load a session to break, save that session, exit out of the application, set the buffer to the next highest setting , re launch application reset audio engine at new buffer setting, load previous saved session and continue loading to break, save again, exit, reset, and so on from 032 thru to 256.
On this occasion tho, on opening the 064 session at the 128 buffer setting, I was confronted with a distinct red performance meter , and any attempt at playing the session instantly thru up a buffer warning. In all of the years I have been using this specific testing methodology, I have never experienced a saved session from a previous buffer setting not being able to be opened and run at the next highest buffer setting. To say it was out of the ordinary was an understatement.
My initial thought was that the audio engine had wigged out , so to double check that I exited out and reset the audio buffers to 032, and then opened the fully loaded session that I had saved @032 , fully expecting the audio engine to still be in a state of being inoperable . Of course that session loaded and played back fine, session is right on the limit, but playing back clean. O.K, exit out , change buffer to 064 , re launch PT9 , load the 032 session , no problem playing back. Close that session and open the even heavier loaded 064 session, plays back no problem , scratch and shake head, close session, shutdown Protools , shift buffer to 128, re launch PT9 , load session saved @ 032, no red performance meter, that a good sign I think to myself , press play, instant cough of the audio engine and buffer warning.. WTF ! ?
I had no choice but to start the 128 testing from a clean slate to see exactly where it was borking. As you can see from the above graph, the results @128 were down 10% from the 064 results , the 256 results did not even match the results I achieved at 032, let alone @064.
These results were defying any logic that I could attributed to how scaling has always increased respective of buffer size. I was starting to feel like I was in a Twilight Zone episode.. :-)
I had noticed on the MD5 results in the previous round that the foot had come off the pedal @128 in regards to scaling, but didn't really think too much about it, the scaling was at least still going in the right direction.
This session however was without saying, going against the norm of what I have experienced with this benching methodology in the 5 1/2 years I have been using it , and was not witnessed with PT9 using the DAE audio engine using the M-Audio interface.
I had witnessed odd behavior with plugin's on OSX during the testing , so I couldn't totally discount this being a strange buffering issue with the plugin itself, but because I hadn't experienced the issue with the M-Audio/DAE configuration , it was less likely.
Despite the U turn, comparative performance was still in favor of Windows across the board.
The next round with the mPressor would either confirm it as being an exception, or plunge us further down the rabbit hole.
|Round 4 : DAWbench DSP : Elysia mPressor : RME HDSPe AIO :|
We are now down to the
last of the test session runs using the Elysia mPressor.
Protools 9.0 : @032 no comparative was available as that setting was unavailable on OSX, @064 performed better on Windows by 23%, @ 128 performance was better on Windows by 1 plugin ( .06%) @ 256 performed better on Windows by 2%
Nuendo 5.0 : @ 032 - performed better on Win7 by 237%
, @ 064 - performed better on Win7 by 98% , @128 - performed better
on Win7 by 71%,
StudioOne 1.5 : @ 032 - performed better on Win7 by 136% , @ 064 - performed better on Win7 by 96% , @128 - performed better on Win7 by 70%, @ 256 - performed better on Win7 by 97 %
Reaper 3.6 : @ 032 - performed better
on Win7 by 61%, @ 064 - performed better on Win7 by 52%, @128 - performed
better on Win7 by 48%,
On OSX, the results for Protool 9 were again very similar to the performance of the DAE M-Audio results, very consistent across the board and easily outperforming all of the other applications @ the available buffer settings.
The Windows results again showed that weird U turn @ 128 buffers , and again the results for 256 did not reach the results achieved @ 032 samples . At least we had some consistency, but that was not exactly a positive in this case.
The overall results on Windows again also showed a distinct advantage to the VST driven applications across the board, as witnessed with the results with DAE/M-Audio
The comparative performance again favored Windows , not by a huge amount but measurable up to 23% at the sweet spot of 064.
With 2 of the 3 sessions displaying an almost identical
performance dip @ 128 , it would be harder to shift the blame to the
respective plugin's. Of course it would have been more conclusive if
all 3 displayed the same behavior , but there is still more likelihood
that it is not plugin based , but more so an ASIO related issue, and
could even be specific to RME HDSPe.
Well this round was definitely interesting, not only was it the first time that Protools was able to be tested in a direct comparative head to head with the preferred reference hardware , but also using the previously alien protocols of ASIO/ Core Audio.
Overall Protools 9 has been pretty impressive , not only has it maintained far better cross platform comparative performance using both DAE and ASIO/Core Audio, but it has performed admirably against the more established open hardware DAW applications and plugin formats. I think its safe to say that RTAS - for plugin's at least, is equally as efficient as VST in most cases on Windows, and more efficient on OSX. That is definitely a turn around to the pre conceived ideas I had coming into this testing.
So plenty of positives, but I just a can't help but think that AVID missed an elephant in the room with the ASIO performance dip at 128 samples on Windows. Now some may think that in Real World application the performance dip would be irrelevant as most would not push their systems to the point were it would be even noticeable and that may be the case for larger i7 based systems for example , but move the working environment to a laptop and it could throw up some odd scenarios very quickly.
In practicality, if you pushed a system close to break @ 064 , moving the buffer up a notch to 128 to buy some more headroom- as would be normal practice , it would in fact be more detrimental
Also the the other scenario of someone working @128, would have more headroom moving down in buffer size. It defies logic and it could point to some other under bonnet gremlins unless addressed .
I have forwarded the information to AVID development and will keep you informed if anything eventuates.
That rounds it off for 2010, I'll be back in 2011 with the some more fun and games.