Quantcast

Extrem High IOPS

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Extrem High IOPS

Sascha Effert

Hello,

 

I have 16 HDDs (not SSDs!) behind a LSI RAID Controller. The disks are old Seagate Contellation, 7200 RPM, 500 GB capacity. I configured this 16 disks as a RAID 10 and deactivated any cache of the controller (in any case: the controller has only 512 MB of cache). My host has 16 GB RAM, my virtual disk has about 7 TB capacity. My OS is Windows 2008 R2 SP1.

 

I started measuring the disk using IOMeter 1.1RC, also tried 2008-06-22-rc2. I am using the disk without filesystem to generate random reads with 4 K blocksize. I am using one worker with one outstanding IO. As sector size I tried 0, 500000000, 5000000000 and 10000000000.

 

My Problem is, that I get about 8000 IOPS with this workload. The average response time is 0,1225 ms. I do not believe that this is possible using normal disks. Using more outstanding IOs I can also get 17000 IOPS.

 

Can anybody give me a hint, what is going wrong? Perhaps I should offer this high-performance disk array at ebay… J

 

Bests

 

Sascha

 


------------------------------------------------------------------------------
Keep yourself connected to Go Parallel:
INSIGHTS What's next for parallel hardware, programming and related areas?
Interviews and blogs by thought leaders keep you ahead of the curve.
http://goparallel.sourceforge.net
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Extrem High IOPS

Sascha Effert-2

Hello,

 

16 GB RAM in the host machine, 512 MB Cache in the controller, 16 MB cache in each single disk.The complete volume takes 7 TB of data and I think I distributed the data over the whole virtual volume (at least using 0 sectors should do this…). I did not use files or a file system, I worked directly on the disk. So, I do not think, that I cache should be the problem, even if all cache is used…

 

I also made a test with a file system, but here I used only about 250 GB as file size, resulting in 57000 IOPS (with 8 workes with 16 outstanding IOs).

 

Bests

 

Sascha

 

Von: Coyne, Kate [mailto:[hidden email]]
Gesendet: Mittwoch, 28. November 2012 18:28
An: Sascha Effert (externe Mail); [hidden email]
Betreff: RE: [Iometer-user] Extrem High IOPS

 

Sascha,

Off the bat looks like cache hits to me….how much cache is on your controller and how big are your files? Does this exact same test on the older version of IOmeter behave the same way?

Cheers,
Kate

 

From: Sascha Effert (externe Mail) [[hidden email]]
Sent: Wednesday, November 28, 2012 12:25 PM
To: [hidden email]
Subject: [Iometer-user] Extrem High IOPS

 

Hello,

 

I have 16 HDDs (not SSDs!) behind a LSI RAID Controller. The disks are old Seagate Contellation, 7200 RPM, 500 GB capacity. I configured this 16 disks as a RAID 10 and deactivated any cache of the controller (in any case: the controller has only 512 MB of cache). My host has 16 GB RAM, my virtual disk has about 7 TB capacity. My OS is Windows 2008 R2 SP1.

 

I started measuring the disk using IOMeter 1.1RC, also tried 2008-06-22-rc2. I am using the disk without filesystem to generate random reads with 4 K blocksize. I am using one worker with one outstanding IO. As sector size I tried 0, 500000000, 5000000000 and 10000000000.

 

My Problem is, that I get about 8000 IOPS with this workload. The average response time is 0,1225 ms. I do not believe that this is possible using normal disks. Using more outstanding IOs I can also get 17000 IOPS.

 

Can anybody give me a hint, what is going wrong? Perhaps I should offer this high-performance disk array at ebay… J

 

Bests

 

Sascha

 


------------------------------------------------------------------------------
Keep yourself connected to Go Parallel:
INSIGHTS What's next for parallel hardware, programming and related areas?
Interviews and blogs by thought leaders keep you ahead of the curve.
http://goparallel.sourceforge.net
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Extrem High IOPS

Sascha Effert-2

Hello,

 

where should I find the file if there is no filesystem? Does IOMeter support driver greater 2 TB? I am using 64 Bit version on 64 Bit Windows, so I think this should be no problem, but I am not sure.

 

I any case: thanks for helping during your vacation! J

 

Bests

 

Sascha

 

Von: Coyne, Kate [mailto:[hidden email]]
Gesendet: Mittwoch, 28. November 2012 18:37
An: Sascha Effert; Sascha Effert (externe Mail); [hidden email]
Betreff: RE: [Iometer-user] Extrem High IOPS

 

Sascha,

Sorry you did mention that you used raw disks not file system!  Mmmm yes that will not fit into cache ..the only other way I’ve seen weird numbers like that is when I had a 0 size iobw.tst file by mistake build and then ran the test on it again(thinking it was large).   Check the size of the iowb.tst file that is created(and I believe this is done for raw disks too) just in case…otherwise you will need to talk to the wise Iometer experts.:>

Cheers,
Kate

 

From: Sascha Effert [[hidden email]]
Sent: Wednesday, November 28, 2012 12:34 PM
To: Coyne, Kate; Sascha Effert (externe Mail); [hidden email]
Subject: AW: [Iometer-user] Extrem High IOPS

 

Hello,

 

16 GB RAM in the host machine, 512 MB Cache in the controller, 16 MB cache in each single disk.The complete volume takes 7 TB of data and I think I distributed the data over the whole virtual volume (at least using 0 sectors should do this…). I did not use files or a file system, I worked directly on the disk. So, I do not think, that I cache should be the problem, even if all cache is used…

 

I also made a test with a file system, but here I used only about 250 GB as file size, resulting in 57000 IOPS (with 8 workes with 16 outstanding IOs).

 

Bests

 

Sascha

 

Von: Coyne, Kate [[hidden email]]
Gesendet: Mittwoch, 28. November 2012 18:28
An: Sascha Effert (externe Mail); [hidden email]
Betreff: RE: [Iometer-user] Extrem High IOPS

 

Sascha,

Off the bat looks like cache hits to me….how much cache is on your controller and how big are your files? Does this exact same test on the older version of IOmeter behave the same way?

Cheers,
Kate

 

From: Sascha Effert (externe Mail) [[hidden email]]
Sent: Wednesday, November 28, 2012 12:25 PM
To: [hidden email]
Subject: [Iometer-user] Extrem High IOPS

 

Hello,

 

I have 16 HDDs (not SSDs!) behind a LSI RAID Controller. The disks are old Seagate Contellation, 7200 RPM, 500 GB capacity. I configured this 16 disks as a RAID 10 and deactivated any cache of the controller (in any case: the controller has only 512 MB of cache). My host has 16 GB RAM, my virtual disk has about 7 TB capacity. My OS is Windows 2008 R2 SP1.

 

I started measuring the disk using IOMeter 1.1RC, also tried 2008-06-22-rc2. I am using the disk without filesystem to generate random reads with 4 K blocksize. I am using one worker with one outstanding IO. As sector size I tried 0, 500000000, 5000000000 and 10000000000.

 

My Problem is, that I get about 8000 IOPS with this workload. The average response time is 0,1225 ms. I do not believe that this is possible using normal disks. Using more outstanding IOs I can also get 17000 IOPS.

 

Can anybody give me a hint, what is going wrong? Perhaps I should offer this high-performance disk array at ebay… J

 

Bests

 

Sascha

 


------------------------------------------------------------------------------
Keep yourself connected to Go Parallel:
INSIGHTS What's next for parallel hardware, programming and related areas?
Interviews and blogs by thought leaders keep you ahead of the curve.
http://goparallel.sourceforge.net
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Extrem High IOPS

Vedran Degoricija
Raw disks access does not rely on a iobw.tst file -- you'd need a filesystem for that.

In general, IOmeter does not open files or devices in cached mode on Windows, so filesystem caching should never be an issue. I wonder if this is an artifact of some LSI RAID optimization. What happens with writes for your FS and raw cases?

Regards,
Ved


From: Sascha Effert <[hidden email]>
To: "'Coyne, Kate'" <[hidden email]>; Sascha Effert (externe Mail) <[hidden email]>; [hidden email]
Sent: Wednesday, November 28, 2012 9:43 AM
Subject: Re: [Iometer-user] Extrem High IOPS

Hello,
 
where should I find the file if there is no filesystem? Does IOMeter support driver greater 2 TB? I am using 64 Bit version on 64 Bit Windows, so I think this should be no problem, but I am not sure.
 
I any case: thanks for helping during your vacation! J
 
Bests
 
Sascha
 
Von: Coyne, Kate [mailto:[hidden email]]
Gesendet: Mittwoch, 28. November 2012 18:37
An: Sascha Effert; Sascha Effert (externe Mail); [hidden email]
Betreff: RE: [Iometer-user] Extrem High IOPS
 
Sascha,
Sorry you did mention that you used raw disks not file system!  Mmmm yes that will not fit into cache ..the only other way I’ve seen weird numbers like that is when I had a 0 size iobw.tst file by mistake build and then ran the test on it again(thinking it was large).   Check the size of the iowb.tst file that is created(and I believe this is done for raw disks too) just in case…otherwise you will need to talk to the wise Iometer experts.:>
Cheers,
Kate
 
From: Sascha Effert [[hidden email]]
Sent: Wednesday, November 28, 2012 12:34 PM
To: Coyne, Kate; Sascha Effert (externe Mail); [hidden email]
Subject: AW: [Iometer-user] Extrem High IOPS
 
Hello,
 
16 GB RAM in the host machine, 512 MB Cache in the controller, 16 MB cache in each single disk.The complete volume takes 7 TB of data and I think I distributed the data over the whole virtual volume (at least using 0 sectors should do this…). I did not use files or a file system, I worked directly on the disk. So, I do not think, that I cache should be the problem, even if all cache is used…
 
I also made a test with a file system, but here I used only about 250 GB as file size, resulting in 57000 IOPS (with 8 workes with 16 outstanding IOs).
 
Bests
 
Sascha
 
Von: Coyne, Kate [[hidden email]]
Gesendet: Mittwoch, 28. November 2012 18:28
An: Sascha Effert (externe Mail); [hidden email]
Betreff: RE: [Iometer-user] Extrem High IOPS
 
Sascha,
Off the bat looks like cache hits to me….how much cache is on your controller and how big are your files? Does this exact same test on the older version of IOmeter behave the same way?
Cheers,
Kate
 
From: Sascha Effert (externe Mail) [[hidden email]]
Sent: Wednesday, November 28, 2012 12:25 PM
To: [hidden email]
Subject: [Iometer-user] Extrem High IOPS
 
Hello,
 
I have 16 HDDs (not SSDs!) behind a LSI RAID Controller. The disks are old Seagate Contellation, 7200 RPM, 500 GB capacity. I configured this 16 disks as a RAID 10 and deactivated any cache of the controller (in any case: the controller has only 512 MB of cache). My host has 16 GB RAM, my virtual disk has about 7 TB capacity. My OS is Windows 2008 R2 SP1.
 
I started measuring the disk using IOMeter 1.1RC, also tried 2008-06-22-rc2. I am using the disk without filesystem to generate random reads with 4 K blocksize. I am using one worker with one outstanding IO. As sector size I tried 0, 500000000, 5000000000 and 10000000000.
 
My Problem is, that I get about 8000 IOPS with this workload. The average response time is 0,1225 ms. I do not believe that this is possible using normal disks. Using more outstanding IOs I can also get 17000 IOPS.
 
Can anybody give me a hint, what is going wrong? Perhaps I should offer this high-performance disk array at ebay… J
 
Bests
 
Sascha
 

------------------------------------------------------------------------------
Keep yourself connected to Go Parallel:
INSIGHTS What's next for parallel hardware, programming and related areas?
Interviews and blogs by thought leaders keep you ahead of the curve.
http://goparallel.sourceforge.net
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user



------------------------------------------------------------------------------
Keep yourself connected to Go Parallel:
INSIGHTS What's next for parallel hardware, programming and related areas?
Interviews and blogs by thought leaders keep you ahead of the curve.
http://goparallel.sourceforge.net
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Extrem High IOPS

Sascha Effert-2

Hello,

 

i started a test running on a file system using the full capacity. Unfortunately it will take until tommow to get results, because the preparation of a 7 TB file takes some time…. I will send the results when they are available.

 

Bests

 

Sascha

 

Von: Vedran Degoricija [mailto:[hidden email]]
Gesendet: Mittwoch, 28. November 2012 21:52
An: Sascha Effert; 'Coyne, Kate'; Sascha Effert (externe Mail); [hidden email]
Betreff: Re: [Iometer-user] Extrem High IOPS

 

Raw disks access does not rely on a iobw.tst file -- you'd need a filesystem for that.

 

In general, IOmeter does not open files or devices in cached mode on Windows, so filesystem caching should never be an issue. I wonder if this is an artifact of some LSI RAID optimization. What happens with writes for your FS and raw cases?

 

Regards,

Ved

 


From: Sascha Effert <[hidden email]>
To: "'Coyne, Kate'" <[hidden email]>; Sascha Effert (externe Mail) <[hidden email]>; [hidden email]
Sent: Wednesday, November 28, 2012 9:43 AM
Subject: Re: [Iometer-user] Extrem High IOPS

 

Hello,

 

where should I find the file if there is no filesystem? Does IOMeter support driver greater 2 TB? I am using 64 Bit version on 64 Bit Windows, so I think this should be no problem, but I am not sure.

 

I any case: thanks for helping during your vacation! J

 

Bests

 

Sascha

 

Von: Coyne, Kate [[hidden email]]
Gesendet: Mittwoch, 28. November 2012 18:37
An: Sascha Effert; Sascha Effert (externe Mail); [hidden email]
Betreff: RE: [Iometer-user] Extrem High IOPS

 

Sascha,

Sorry you did mention that you used raw disks not file system!  Mmmm yes that will not fit into cache ..the only other way I’ve seen weird numbers like that is when I had a 0 size iobw.tst file by mistake build and then ran the test on it again(thinking it was large).   Check the size of the iowb.tst file that is created(and I believe this is done for raw disks too) just in case…otherwise you will need to talk to the wise Iometer experts.:>

Cheers,
Kate

 

From: Sascha Effert [[hidden email]]
Sent: Wednesday, November 28, 2012 12:34 PM
To: Coyne, Kate; Sascha Effert (externe Mail); [hidden email]
Subject: AW: [Iometer-user] Extrem High IOPS

 

Hello,

 

16 GB RAM in the host machine, 512 MB Cache in the controller, 16 MB cache in each single disk.The complete volume takes 7 TB of data and I think I distributed the data over the whole virtual volume (at least using 0 sectors should do this…). I did not use files or a file system, I worked directly on the disk. So, I do not think, that I cache should be the problem, even if all cache is used…

 

I also made a test with a file system, but here I used only about 250 GB as file size, resulting in 57000 IOPS (with 8 workes with 16 outstanding IOs).

 

Bests

 

Sascha

 

Von: Coyne, Kate [[hidden email]]
Gesendet: Mittwoch, 28. November 2012 18:28
An: Sascha Effert (externe Mail); [hidden email]
Betreff: RE: [Iometer-user] Extrem High IOPS

 

Sascha,

Off the bat looks like cache hits to me….how much cache is on your controller and how big are your files? Does this exact same test on the older version of IOmeter behave the same way?

Cheers,
Kate

 

From: Sascha Effert (externe Mail) [[hidden email]]
Sent: Wednesday, November 28, 2012 12:25 PM
To: [hidden email]
Subject: [Iometer-user] Extrem High IOPS

 

Hello,

 

I have 16 HDDs (not SSDs!) behind a LSI RAID Controller. The disks are old Seagate Contellation, 7200 RPM, 500 GB capacity. I configured this 16 disks as a RAID 10 and deactivated any cache of the controller (in any case: the controller has only 512 MB of cache). My host has 16 GB RAM, my virtual disk has about 7 TB capacity. My OS is Windows 2008 R2 SP1.

 

I started measuring the disk using IOMeter 1.1RC, also tried 2008-06-22-rc2. I am using the disk without filesystem to generate random reads with 4 K blocksize. I am using one worker with one outstanding IO. As sector size I tried 0, 500000000, 5000000000 and 10000000000.

 

My Problem is, that I get about 8000 IOPS with this workload. The average response time is 0,1225 ms. I do not believe that this is possible using normal disks. Using more outstanding IOs I can also get 17000 IOPS.

 

Can anybody give me a hint, what is going wrong? Perhaps I should offer this high-performance disk array at ebay… J

 

Bests

 

Sascha

 


------------------------------------------------------------------------------
Keep yourself connected to Go Parallel:
INSIGHTS What's next for parallel hardware, programming and related areas?
Interviews and blogs by thought leaders keep you ahead of the curve.
http://goparallel.sourceforge.net
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user


------------------------------------------------------------------------------
Keep yourself connected to Go Parallel:
VERIFY Test and improve your parallel project with help from experts
and peers. http://goparallel.sourceforge.net
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Extrem High IOPS

Sascha Effert-2

Hello,

 

I have results for full access on file level. Here I get about 90 IOPS with 1 outstanding IO, about 900 with one worker and one outstanding IO and about 2000 with 8 workers and 16 Outstanding IOs (each worker). So it is near to what I supposed to get.

 

I do not understand, why the tests did not work when writing directly to disk without file system. Is there anything left I made wrong? Could this be a failure somewhere in ioMeter?

 

My problem is, that the tests using the filesystem are a big problem, just to get a short idea of what my system can do. The preparation of the 7 TB took about 14 hours…

 

Bests

 

 

Sascha

 

Von: Sascha Effert [mailto:[hidden email]]
Gesendet: Donnerstag, 29. November 2012 13:36
An: 'Vedran Degoricija'; 'Coyne, Kate'; 'Sascha Effert (externe Mail)'; [hidden email]
Betreff: AW: [Iometer-user] Extrem High IOPS

 

Hello,

 

i started a test running on a file system using the full capacity. Unfortunately it will take until tommow to get results, because the preparation of a 7 TB file takes some time…. I will send the results when they are available.

 

Bests

 

Sascha

 

Von: Vedran Degoricija [[hidden email]]
Gesendet: Mittwoch, 28. November 2012 21:52
An: Sascha Effert; 'Coyne, Kate'; Sascha Effert (externe Mail); [hidden email]
Betreff: Re: [Iometer-user] Extrem High IOPS

 

Raw disks access does not rely on a iobw.tst file -- you'd need a filesystem for that.

 

In general, IOmeter does not open files or devices in cached mode on Windows, so filesystem caching should never be an issue. I wonder if this is an artifact of some LSI RAID optimization. What happens with writes for your FS and raw cases?

 

Regards,

Ved

 


From: Sascha Effert <[hidden email]>
To: "'Coyne, Kate'" <[hidden email]>; Sascha Effert (externe Mail) <[hidden email]>; [hidden email]
Sent: Wednesday, November 28, 2012 9:43 AM
Subject: Re: [Iometer-user] Extrem High IOPS

 

Hello,

 

where should I find the file if there is no filesystem? Does IOMeter support driver greater 2 TB? I am using 64 Bit version on 64 Bit Windows, so I think this should be no problem, but I am not sure.

 

I any case: thanks for helping during your vacation! J

 

Bests

 

Sascha

 

Von: Coyne, Kate [[hidden email]]
Gesendet: Mittwoch, 28. November 2012 18:37
An: Sascha Effert; Sascha Effert (externe Mail); [hidden email]
Betreff: RE: [Iometer-user] Extrem High IOPS

 

Sascha,

Sorry you did mention that you used raw disks not file system!  Mmmm yes that will not fit into cache ..the only other way I’ve seen weird numbers like that is when I had a 0 size iobw.tst file by mistake build and then ran the test on it again(thinking it was large).   Check the size of the iowb.tst file that is created(and I believe this is done for raw disks too) just in case…otherwise you will need to talk to the wise Iometer experts.:>

Cheers,
Kate

 

From: Sascha Effert [[hidden email]]
Sent: Wednesday, November 28, 2012 12:34 PM
To: Coyne, Kate; Sascha Effert (externe Mail); [hidden email]
Subject: AW: [Iometer-user] Extrem High IOPS

 

Hello,

 

16 GB RAM in the host machine, 512 MB Cache in the controller, 16 MB cache in each single disk.The complete volume takes 7 TB of data and I think I distributed the data over the whole virtual volume (at least using 0 sectors should do this…). I did not use files or a file system, I worked directly on the disk. So, I do not think, that I cache should be the problem, even if all cache is used…

 

I also made a test with a file system, but here I used only about 250 GB as file size, resulting in 57000 IOPS (with 8 workes with 16 outstanding IOs).

 

Bests

 

Sascha

 

Von: Coyne, Kate [[hidden email]]
Gesendet: Mittwoch, 28. November 2012 18:28
An: Sascha Effert (externe Mail); [hidden email]
Betreff: RE: [Iometer-user] Extrem High IOPS

 

Sascha,

Off the bat looks like cache hits to me….how much cache is on your controller and how big are your files? Does this exact same test on the older version of IOmeter behave the same way?

Cheers,
Kate

 

From: Sascha Effert (externe Mail) [[hidden email]]
Sent: Wednesday, November 28, 2012 12:25 PM
To: [hidden email]
Subject: [Iometer-user] Extrem High IOPS

 

Hello,

 

I have 16 HDDs (not SSDs!) behind a LSI RAID Controller. The disks are old Seagate Contellation, 7200 RPM, 500 GB capacity. I configured this 16 disks as a RAID 10 and deactivated any cache of the controller (in any case: the controller has only 512 MB of cache). My host has 16 GB RAM, my virtual disk has about 7 TB capacity. My OS is Windows 2008 R2 SP1.

 

I started measuring the disk using IOMeter 1.1RC, also tried 2008-06-22-rc2. I am using the disk without filesystem to generate random reads with 4 K blocksize. I am using one worker with one outstanding IO. As sector size I tried 0, 500000000, 5000000000 and 10000000000.

 

My Problem is, that I get about 8000 IOPS with this workload. The average response time is 0,1225 ms. I do not believe that this is possible using normal disks. Using more outstanding IOs I can also get 17000 IOPS.

 

Can anybody give me a hint, what is going wrong? Perhaps I should offer this high-performance disk array at ebay… J

 

Bests

 

Sascha

 


------------------------------------------------------------------------------
Keep yourself connected to Go Parallel:
INSIGHTS What's next for parallel hardware, programming and related areas?
Interviews and blogs by thought leaders keep you ahead of the curve.
http://goparallel.sourceforge.net
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user


------------------------------------------------------------------------------
Keep yourself connected to Go Parallel:
TUNE You got it built. Now make it sing. Tune shows you how.
http://goparallel.sourceforge.net
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Extrem High IOPS

Vedran Degoricija
Hi Sacha,

There should definitely not be such a huge discrepancy between filesystem and raw device access. You should try raw access again to see if the behavior has changed. Always try to perform some writes to the raw device before you do the reads.

Also, there is no need to prepare the entire filesystem to gauge performance. A few gigabytes is the max you should need.

Good luck.
Ved


From: Sascha Effert <[hidden email]>
To: 'Vedran Degoricija' <[hidden email]>; "'Coyne, Kate'" <[hidden email]>; Sascha Effert (externe Mail) <[hidden email]>; [hidden email]
Sent: Friday, November 30, 2012 12:16 AM
Subject: Re: [Iometer-user] Extrem High IOPS

Hello,
 
I have results for full access on file level. Here I get about 90 IOPS with 1 outstanding IO, about 900 with one worker and one outstanding IO and about 2000 with 8 workers and 16 Outstanding IOs (each worker). So it is near to what I supposed to get.
 
I do not understand, why the tests did not work when writing directly to disk without file system. Is there anything left I made wrong? Could this be a failure somewhere in ioMeter?
 
My problem is, that the tests using the filesystem are a big problem, just to get a short idea of what my system can do. The preparation of the 7 TB took about 14 hours…
 
Bests
 
 
Sascha
 
Von: Sascha Effert [mailto:[hidden email]]
Gesendet: Donnerstag, 29. November 2012 13:36
An: 'Vedran Degoricija'; 'Coyne, Kate'; 'Sascha Effert (externe Mail)'; [hidden email]
Betreff: AW: [Iometer-user] Extrem High IOPS
 
Hello,
 
i started a test running on a file system using the full capacity. Unfortunately it will take until tommow to get results, because the preparation of a 7 TB file takes some time…. I will send the results when they are available.
 
Bests
 
Sascha
 
Von: Vedran Degoricija [[hidden email]]
Gesendet: Mittwoch, 28. November 2012 21:52
An: Sascha Effert; 'Coyne, Kate'; Sascha Effert (externe Mail); [hidden email]
Betreff: Re: [Iometer-user] Extrem High IOPS
 
Raw disks access does not rely on a iobw.tst file -- you'd need a filesystem for that.
 
In general, IOmeter does not open files or devices in cached mode on Windows, so filesystem caching should never be an issue. I wonder if this is an artifact of some LSI RAID optimization. What happens with writes for your FS and raw cases?
 
Regards,
Ved
 

From: Sascha Effert <[hidden email]>
To: "'Coyne, Kate'" <[hidden email]>; Sascha Effert (externe Mail) <[hidden email]>; [hidden email]
Sent: Wednesday, November 28, 2012 9:43 AM
Subject: Re: [Iometer-user] Extrem High IOPS
 
Hello,
 
where should I find the file if there is no filesystem? Does IOMeter support driver greater 2 TB? I am using 64 Bit version on 64 Bit Windows, so I think this should be no problem, but I am not sure.
 
I any case: thanks for helping during your vacation! J
 
Bests
 
Sascha
 
Von: Coyne, Kate [[hidden email]]
Gesendet: Mittwoch, 28. November 2012 18:37
An: Sascha Effert; Sascha Effert (externe Mail); [hidden email]
Betreff: RE: [Iometer-user] Extrem High IOPS
 
Sascha,
Sorry you did mention that you used raw disks not file system!  Mmmm yes that will not fit into cache ..the only other way I’ve seen weird numbers like that is when I had a 0 size iobw.tst file by mistake build and then ran the test on it again(thinking it was large).   Check the size of the iowb.tst file that is created(and I believe this is done for raw disks too) just in case…otherwise you will need to talk to the wise Iometer experts.:>
Cheers,
Kate
 
From: Sascha Effert [[hidden email]]
Sent: Wednesday, November 28, 2012 12:34 PM
To: Coyne, Kate; Sascha Effert (externe Mail); [hidden email]
Subject: AW: [Iometer-user] Extrem High IOPS
 
Hello,
 
16 GB RAM in the host machine, 512 MB Cache in the controller, 16 MB cache in each single disk.The complete volume takes 7 TB of data and I think I distributed the data over the whole virtual volume (at least using 0 sectors should do this…). I did not use files or a file system, I worked directly on the disk. So, I do not think, that I cache should be the problem, even if all cache is used…
 
I also made a test with a file system, but here I used only about 250 GB as file size, resulting in 57000 IOPS (with 8 workes with 16 outstanding IOs).
 
Bests
 
Sascha
 
Von: Coyne, Kate [[hidden email]]
Gesendet: Mittwoch, 28. November 2012 18:28
An: Sascha Effert (externe Mail); [hidden email]
Betreff: RE: [Iometer-user] Extrem High IOPS
 
Sascha,
Off the bat looks like cache hits to me….how much cache is on your controller and how big are your files? Does this exact same test on the older version of IOmeter behave the same way?
Cheers,
Kate
 
From: Sascha Effert (externe Mail) [[hidden email]]
Sent: Wednesday, November 28, 2012 12:25 PM
To: [hidden email]
Subject: [Iometer-user] Extrem High IOPS
 
Hello,
 
I have 16 HDDs (not SSDs!) behind a LSI RAID Controller. The disks are old Seagate Contellation, 7200 RPM, 500 GB capacity. I configured this 16 disks as a RAID 10 and deactivated any cache of the controller (in any case: the controller has only 512 MB of cache). My host has 16 GB RAM, my virtual disk has about 7 TB capacity. My OS is Windows 2008 R2 SP1.
 
I started measuring the disk using IOMeter 1.1RC, also tried 2008-06-22-rc2. I am using the disk without filesystem to generate random reads with 4 K blocksize. I am using one worker with one outstanding IO. As sector size I tried 0, 500000000, 5000000000 and 10000000000.
 
My Problem is, that I get about 8000 IOPS with this workload. The average response time is 0,1225 ms. I do not believe that this is possible using normal disks. Using more outstanding IOs I can also get 17000 IOPS.
 
Can anybody give me a hint, what is going wrong? Perhaps I should offer this high-performance disk array at ebay… J
 
Bests
 
Sascha
 

------------------------------------------------------------------------------
Keep yourself connected to Go Parallel:
INSIGHTS What's next for parallel hardware, programming and related areas?
Interviews and blogs by thought leaders keep you ahead of the curve.
http://goparallel.sourceforge.net
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user

------------------------------------------------------------------------------
Keep yourself connected to Go Parallel:
TUNE You got it built. Now make it sing. Tune shows you how.
http://goparallel.sourceforge.net
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user



------------------------------------------------------------------------------
Keep yourself connected to Go Parallel:
TUNE You got it built. Now make it sing. Tune shows you how.
http://goparallel.sourceforge.net
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Extrem High IOPS

Diego Alonso
I too have the same problem.  I only get realistic numbers when testing with raw disk.  The moment I format it the numbers are way too high.  For instance when I work with a raw disk on my iSCSI setup I get about 6000 IOPs and 200Mb/s which seems about right given my 2x1GB NICs and EqualLogic SAN.

I then format the drive and create iobw.tst file using createtestfile.exe.  when I re-run tests I get 20,000 IOPs and 800Mb/s.  I tried different size iobw.tst files using from 100GB to 800GB and numbers are always crazy.

I have other systems that I need to test but down have raw disks.  Anybody have any idea how to "fix" this?

Thanks,
Diego


Vedran Degoricija wrote
Hi Sacha,

There should definitely not be such a huge discrepancy between filesystem and raw device access. You should try raw access again to see if the behavior has changed. Always try to perform some writes to the raw device before you do the reads.

Also, there is no need to prepare the entire filesystem to gauge performance. A few gigabytes is the max you should need.

Good luck.
Ved



>________________________________
> From: Sascha Effert <fermat@mail.uni-paderborn.de>
>To: 'Vedran Degoricija' <vedrand@yahoo.com>; "'Coyne, Kate'" <Kate.Coyne@stratus.com>; Sascha Effert (externe Mail) <fermat@uni-paderborn.de>; iometer-user@lists.sourceforge.net
>Sent: Friday, November 30, 2012 12:16 AM
>Subject: Re: [Iometer-user] Extrem High IOPS
>
>
>Hello,

>I have results for full access on file level. Here I get about 90 IOPS with 1 outstanding IO, about 900 with one worker and one outstanding IO and about 2000 with 8 workers and 16 Outstanding IOs (each worker). So it is near to what I supposed to get.

>I do not understand, why the tests did not work when writing directly to disk without file system. Is there anything left I made wrong? Could this be a failure somewhere in ioMeter?

>My problem is, that the tests using the filesystem are a big problem, just to get a short idea of what my system can do. The preparation of the 7 TB took about 14 hours…

>Bests


>Sascha

>Von:Sascha Effert [mailto:fermat@mail.uni-paderborn.de]
>Gesendet: Donnerstag, 29. November 2012 13:36
>An: 'Vedran Degoricija'; 'Coyne, Kate'; 'Sascha Effert (externe Mail)'; iometer-user@lists.sourceforge.net
>Betreff: AW: [Iometer-user] Extrem High IOPS

>Hello,

>i started a test running on a file system using the full capacity. Unfortunately it will take until tommow to get results, because the preparation of a 7 TB file takes some time…. I will send the results when they are available.

>Bests

>Sascha

>Von:Vedran Degoricija [mailto:vedrand@yahoo.com]
>Gesendet: Mittwoch, 28. November 2012 21:52
>An: Sascha Effert; 'Coyne, Kate'; Sascha Effert (externe Mail); iometer-user@lists.sourceforge.net
>Betreff: Re: [Iometer-user] Extrem High IOPS

>Raw disks access does not rely on a iobw.tst file -- you'd need a filesystem for that.

>In general, IOmeter does not open files or devices in cached mode on Windows, so filesystem caching should never be an issue. I wonder if this is an artifact of some LSI RAID optimization. What happens with writes for your FS and raw cases?

>Regards,
>Ved

>>
>>________________________________
>>
>>From:Sascha Effert <fermat@mail.uni-paderborn.de>
>>To: "'Coyne, Kate'" <Kate.Coyne@stratus.com>; Sascha Effert (externe Mail) <fermat@uni-paderborn.de>; iometer-user@lists.sourceforge.net
>>Sent: Wednesday, November 28, 2012 9:43 AM
>>Subject: Re: [Iometer-user] Extrem High IOPS
>> 
>>Hello,
>> 
>>where should I find the file if there is no filesystem? Does IOMeter support driver greater 2 TB? I am using 64 Bit version on 64 Bit Windows, so I think this should be no problem, but I am not sure.
>> 
>>I any case: thanks for helping during your vacation! J
>> 
>>Bests
>> 
>>Sascha
>> 
>>Von:Coyne, Kate [mailto:Kate.Coyne@stratus.com]
>>Gesendet: Mittwoch, 28. November 2012 18:37
>>An: Sascha Effert; Sascha Effert (externe Mail); iometer-user@lists.sourceforge.net
>>Betreff: RE: [Iometer-user] Extrem High IOPS
>> 
>>Sascha,
>>Sorry you did mention that you used raw disks not file system!  Mmmm yes that will not fit into cache ..the only other way I’ve seen weird numbers like that is when I had a 0 size iobw.tst file by mistake build and then ran the test on it again(thinking it was large).   Check the size of the iowb.tst file that is created(and I believe this is done for raw disks too) just in case…otherwise you will need to talk to the wise Iometer experts.:>
>>Cheers,
>>Kate
>> 
>>From:Sascha Effert [mailto:fermat@mail.uni-paderborn.de]
>>Sent: Wednesday, November 28, 2012 12:34 PM
>>To: Coyne, Kate; Sascha Effert (externe Mail); iometer-user@lists.sourceforge.net
>>Subject: AW: [Iometer-user] Extrem High IOPS
>> 
>>Hello,
>> 
>>16 GB RAM in the host machine, 512 MB Cache in the controller, 16 MB cache in each single disk.The complete volume takes 7 TB of data and I think I distributed the data over the whole virtual volume (at least using 0 sectors should do this…). I did not use files or a file system, I worked directly on the disk. So, I do not think, that I cache should be the problem, even if all cache is used…
>> 
>>I also made a test with a file system, but here I used only about 250 GB as file size, resulting in 57000 IOPS (with 8 workes with 16 outstanding IOs).
>> 
>>Bests
>> 
>>Sascha
>> 
>>Von:Coyne, Kate [mailto:Kate.Coyne@stratus.com]
>>Gesendet: Mittwoch, 28. November 2012 18:28
>>An: Sascha Effert (externe Mail); iometer-user@lists.sourceforge.net
>>Betreff: RE: [Iometer-user] Extrem High IOPS
>> 
>>Sascha,
>>Off the bat looks like cache hits to me….how much cache is on your controller and how big are your files? Does this exact same test on the older version of IOmeter behave the same way?
>>Cheers,
>>Kate
>> 
>>From:Sascha Effert (externe Mail) [mailto:fermat@uni-paderborn.de]
>>Sent: Wednesday, November 28, 2012 12:25 PM
>>To: iometer-user@lists.sourceforge.net
>>Subject: [Iometer-user] Extrem High IOPS
>> 
>>Hello,
>> 
>>I have 16 HDDs (not SSDs!) behind a LSI RAID Controller. The disks are old Seagate Contellation, 7200 RPM, 500 GB capacity. I configured this 16 disks as a RAID 10 and deactivated any cache of the controller (in any case: the controller has only 512 MB of cache). My host has 16 GB RAM, my virtual disk has about 7 TB capacity. My OS is Windows 2008 R2 SP1.
>> 
>>I started measuring the disk using IOMeter 1.1RC, also tried 2008-06-22-rc2. I am using the disk without filesystem to generate random reads with 4 K blocksize. I am using one worker with one outstanding IO. As sector size I tried 0, 500000000, 5000000000 and 10000000000.
>> 
>>My Problem is, that I get about 8000 IOPS with this workload. The average response time is 0,1225 ms. I do not believe that this is possible using normal disks. Using more outstanding IOs I can also get 17000 IOPS.
>> 
>>Can anybody give me a hint, what is going wrong? Perhaps I should offer this high-performance disk array at ebay… J
>> 
>>Bests
>> 
>>Sascha
>> 
>>
>>------------------------------------------------------------------------------
>>Keep yourself connected to Go Parallel:
>>INSIGHTS What's next for parallel hardware, programming and related areas?
>>Interviews and blogs by thought leaders keep you ahead of the curve.
>>http://goparallel.sourceforge.net
>>_______________________________________________
>>Iometer-user mailing list
>>Iometer-user@lists.sourceforge.net
>>https://lists.sourceforge.net/lists/listinfo/iometer-user
>------------------------------------------------------------------------------
>Keep yourself connected to Go Parallel:
>TUNE You got it built. Now make it sing. Tune shows you how.
>http://goparallel.sourceforge.net
>_______________________________________________
>Iometer-user mailing list
>Iometer-user@lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/iometer-user
>
>
>
------------------------------------------------------------------------------
Keep yourself connected to Go Parallel:
TUNE You got it built. Now make it sing. Tune shows you how.
http://goparallel.sourceforge.net
_______________________________________________
Iometer-user mailing list
Iometer-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/iometer-user
Loading...