Non-repeatable results

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Non-repeatable results

Larvoire, Jean-Francois

Hello,

 

We have an unexpected problem with iometer, when measuring performance for a set of clients accessing a server:

Every time we repeat the tests, we get different results. (Sometimes very different!)

(We look at the total MB/s in the .CSV output, in the All,all lines.)

We have eliminated factors like the iometer version (Same problem with the 2006 version and one rebuilt from the SVN head of tree);

Processor load (Both clients and servers have loads below 60% worst case, and usually much lower);

Network load (the test systems are on isolated networks) ;

And HW and/or SW configuration errors (The same problem is observed on two different systems at two different sites).

So we now suspect a test methodology error.

 

Did other people have similar issues?

If so how did you fix it?

 

We typically have 8 clients, running a total of 32 workers.

The clients connect to the servers via 2 to 4 distinct networks.

The workers access iobw.tst files on the server.

We tried a number of distinct workloads, and most have the same non-reproducibility issue.

 

One of the things we’re not sure about is about the use of iobw.tst files:

Is it OK if several client workers access the same iobw.tst at the same time?

Or should every worker access distinct iobw.tst files? (Which requires creating distinct file files for every worker.)

We also wonder about their size. How big should the iobw.tst files be to avoid caching effects?

 

Any other suggestion on what we could try?

 

Jean-François

 

========================================================================

= Jean-François Larvoire                 =========   _/      ===========

= Hewlett-Packard                        =======    _/           =======

= 5 Avenue Raymond Chanas, Eybens        =====     _/_/_/  _/_/_/  =====

= 38053 Grenoble Cedex 9, FRANCE         =====    _/  _/  _/  _/   =====

= Phone: +33 476 14 13 38                =====   _/  _/  _/_/_/    =====

= Fax:   +33 476 14 43 05                =======        _/       =======

= Email: [hidden email]   ==========    _/     ==========

========================================================================

 


------------------------------------------------------------------------------
Own the Future-Intel® Level Up Game Demo Contest 2013
Rise to greatness in Intel's independent game demo contest.
Compete for recognition, cash, and the chance to get your game
on Steam. $5K grand prize plus 10 genre and skill prizes.
Submit your demo by 6/6/13. http://p.sf.net/sfu/intel_levelupd2d
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Non-repeatable results

Vedran Degoricija
Hi Jean-Francois,
 
It sounds like you're using the network to talk to the server? What sort of NICs are you using-1G, 10G? Any individual CPUs near 100% on either the client or the server?
 
Have you tried one client at a time to see if anything changes? Having a dedicated file per worker might be a good thing, else you aren't really simulating anything practical. Depending on your disk subsystem, different files may also perform differently if you're using spinning media.
 
Now, IOmeter should not be affected by filesystem caches on direct attached storage because it issues unbuffered I/O. However, this is not true of a network redirector (at least on Windows), so YMMV.
 
What are the numbers you're seeing?
 
Regards,
Ved

From: "Larvoire, Jean-Francois" <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Tuesday, March 26, 2013 10:16 AM
Subject: [Iometer-user] Non-repeatable results

Hello,
 
We have an unexpected problem with iometer, when measuring performance for a set of clients accessing a server:
Every time we repeat the tests, we get different results. (Sometimes very different!)
(We look at the total MB/s in the .CSV output, in the All,all lines.)
We have eliminated factors like the iometer version (Same problem with the 2006 version and one rebuilt from the SVN head of tree);
Processor load (Both clients and servers have loads below 60% worst case, and usually much lower);
Network load (the test systems are on isolated networks) ;
And HW and/or SW configuration errors (The same problem is observed on two different systems at two different sites).
So we now suspect a test methodology error.
 
Did other people have similar issues?
If so how did you fix it?
 
We typically have 8 clients, running a total of 32 workers.
The clients connect to the servers via 2 to 4 distinct networks.
The workers access iobw.tst files on the server.
We tried a number of distinct workloads, and most have the same non-reproducibility issue.
 
One of the things we’re not sure about is about the use of iobw.tst files:
Is it OK if several client workers access the same iobw.tst at the same time?
Or should every worker access distinct iobw.tst files? (Which requires creating distinct file files for every worker.)
We also wonder about their size. How big should the iobw.tst files be to avoid caching effects?
 
Any other suggestion on what we could try?
 
Jean-François
 
========================================================================
= Jean-François Larvoire                 =========   _/      ===========
= Hewlett-Packard                        =======    _/           =======
= 5 Avenue Raymond Chanas, Eybens        =====     _/_/_/  _/_/_/  =====
= 38053 Grenoble Cedex 9, FRANCE         =====    _/  _/  _/  _/   =====
= Phone: +33 476 14 13 38                =====   _/  _/  _/_/_/    =====
= Fax:   +33 476 14 43 05                =======        _/       =======
= Email: [hidden email]   ==========    _/     ==========
========================================================================
 

------------------------------------------------------------------------------
Own the Future-Intel&reg; Level Up Game Demo Contest 2013
Rise to greatness in Intel's independent game demo contest.
Compete for recognition, cash, and the chance to get your game
on Steam. $5K grand prize plus 10 genre and skill prizes.
Submit your demo by 6/6/13. http://p.sf.net/sfu/intel_levelupd2d
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user



------------------------------------------------------------------------------
Own the Future-Intel&reg; Level Up Game Demo Contest 2013
Rise to greatness in Intel's independent game demo contest.
Compete for recognition, cash, and the chance to get your game
on Steam. $5K grand prize plus 10 genre and skill prizes.
Submit your demo by 6/6/13. http://p.sf.net/sfu/intel_levelupd2d
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Please remove me from the mailing group

koreaseal89

hello,
i don't want to receive these email,
Please remove me from the mailing group, ths.

koreaseal89
 
Date: 2013-03-27 15:04
Subject: Re: [Iometer-user] Non-repeatable results
Hi Jean-Francois,
 
It sounds like you're using the network to talk to the server? What sort of NICs are you using-1G, 10G? Any individual CPUs near 100% on either the client or the server?
 
Have you tried one client at a time to see if anything changes? Having a dedicated file per worker might be a good thing, else you aren't really simulating anything practical. Depending on your disk subsystem, different files may also perform differently if you're using spinning media.
 
Now, IOmeter should not be affected by filesystem caches on direct attached storage because it issues unbuffered I/O. However, this is not true of a network redirector (at least on Windows), so YMMV.
 
What are the numbers you're seeing?
 
Regards,
Ved

From: "Larvoire, Jean-Francois" <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Tuesday, March 26, 2013 10:16 AM
Subject: [Iometer-user] Non-repeatable results

Hello,
 
We have an unexpected problem with iometer, when measuring performance for a set of clients accessing a server:
Every time we repeat the tests, we get different results. (Sometimes very different!)
(We look at the total MB/s in the .CSV output, in the All,all lines.)
We have eliminated factors like the iometer version (Same problem with the 2006 version and one rebuilt from the SVN head of tree);
Processor load (Both clients and servers have loads below 60% worst case, and usually much lower);
Network load (the test systems are on isolated networks) ;
And HW and/or SW configuration errors (The same problem is observed on two different systems at two different sites).
So we now suspect a test methodology error.
 
Did other people have similar issues?
If so how did you fix it?
 
We typically have 8 clients, running a total of 32 workers.
The clients connect to the servers via 2 to 4 distinct networks.
The workers access iobw.tst files on the server.
We tried a number of distinct workloads, and most have the same non-reproducibility issue.
 
One of the things we’re not sure about is about the use of iobw.tst files:
Is it OK if several client workers access the same iobw.tst at the same time?
Or should every worker access distinct iobw.tst files? (Which requires creating distinct file files for every worker.)
We also wonder about their size. How big should the iobw.tst files be to avoid caching effects?
 
Any other suggestion on what we could try?
 
Jean-François
 
========================================================================
= Jean-François Larvoire                 =========   _/      ===========
= Hewlett-Packard                        =======    _/           =======
= 5 Avenue Raymond Chanas, Eybens        =====     _/_/_/  _/_/_/  =====
= 38053 Grenoble Cedex 9, FRANCE         =====    _/  _/  _/  _/   =====
= Phone: +33 476 14 13 38                =====   _/  _/  _/_/_/    =====
= Fax:   +33 476 14 43 05                =======        _/       =======
= Email: [hidden email]   ==========    _/     ==========
========================================================================
 

------------------------------------------------------------------------------
Own the Future-Intel&reg; Level Up Game Demo Contest 2013
Rise to greatness in Intel's independent game demo contest.
Compete for recognition, cash, and the chance to get your game
on Steam. $5K grand prize plus 10 genre and skill prizes.
Submit your demo by 6/6/13. http://p.sf.net/sfu/intel_levelupd2d
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user



------------------------------------------------------------------------------
Own the Future-Intel&reg; Level Up Game Demo Contest 2013
Rise to greatness in Intel's independent game demo contest.
Compete for recognition, cash, and the chance to get your game
on Steam. $5K grand prize plus 10 genre and skill prizes.
Submit your demo by 6/6/13. http://p.sf.net/sfu/intel_levelupd2d
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Loading...