Quantcast

Simulate sequential and strictly round-robing reads from many files simultaneously?

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Simulate sequential and strictly round-robing reads from many files simultaneously?

Greg Sullivan-3
I want to generate a load that consists of sequential reads from seperate areas of the disk. Each area needs to be accessed in a strict round robin fashion. I.e, after the first area has completed a read transfer, the second area should receive one transfer, and then the third, etc etc, and after all areas have been accessed once, the next block from the first area is read, and so on. The reason I want this type of access is to simulate a real time application that performs synchronous, sequential, reads from many files simultaneously. 

I have tried using many workers, with each worker having a different offset, however when I inspect the throughput for individual workers, they are not identical. If it were working the way I want it to, the throughput would be absolutely identical for every worker.

If IOMeter can't do what I want, I'd appreciate suggestions for alternatives.

Thanks,
Greg.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Simulate sequential and strictly round-robing reads from many files simultaneously?

Matt Alsip

Hi Greg,

 

If you have spread the workers out among the entire disk, then you should expect the sequential throughput to vary.  This is because the throughput on outer cylinders of the disk is faster than the throughput of inner cylinders of the disk.  This is a physical characteristic of disk drives, and the behavior even carries through in groups of disks (RAID sets), unless the RAID vendor makes a special effort to alternate disk usage (inner-to-outer vs outer-to-inner).

 

Hope this helps,

 

Regards,

Matt Alsip

Technical Marketing Manager

Dot Hill Systems

 

From: Greg Sullivan [mailto:[hidden email]]
Sent: Thursday, August 09, 2012 6:39 AM
To: [hidden email]
Subject: [Iometer-user] Simulate sequential and strictly round-robing reads from many files simultaneously?

 

I want to generate a load that consists of sequential reads from seperate areas of the disk. Each area needs to be accessed in a strict round robin fashion. I.e, after the first area has completed a read transfer, the second area should receive one transfer, and then the third, etc etc, and after all areas have been accessed once, the next block from the first area is read, and so on. The reason I want this type of access is to simulate a real time application that performs synchronous, sequential, reads from many files simultaneously. 

 

I have tried using many workers, with each worker having a different offset, however when I inspect the throughput for individual workers, they are not identical. If it were working the way I want it to, the throughput would be absolutely identical for every worker.

 

If IOMeter can't do what I want, I'd appreciate suggestions for alternatives.

 

Thanks,

Greg.


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Simulate sequential and strictly round-robing reads from many files simultaneously?

Greg Sullivan-3

Matt,
Imagine there are ten regions, region 1 to region 10. I want IOmeter to read a block from region #1, then from region #2,  then region #3, all the way to #10. When it has read the first block from all regions, it should then read the second block from region #1, region 2, and so on.  If it were to do the io in strict order like this, then the throughput for each worker would be 100% identical.

How can I generate this io pattern?

Regards,
Greg.

On Aug 10, 2012 12:21 AM, "Matt Alsip" <[hidden email]> wrote:

Hi Greg,

 

If you have spread the workers out among the entire disk, then you should expect the sequential throughput to vary.  This is because the throughput on outer cylinders of the disk is faster than the throughput of inner cylinders of the disk.  This is a physical characteristic of disk drives, and the behavior even carries through in groups of disks (RAID sets), unless the RAID vendor makes a special effort to alternate disk usage (inner-to-outer vs outer-to-inner).

 

Hope this helps,

 

Regards,

Matt Alsip

Technical Marketing Manager

Dot Hill Systems

 

From: Greg Sullivan [mailto:[hidden email]]
Sent: Thursday, August 09, 2012 6:39 AM
To: [hidden email]
Subject: [Iometer-user] Simulate sequential and strictly round-robing reads from many files simultaneously?

 

I want to generate a load that consists of sequential reads from seperate areas of the disk. Each area needs to be accessed in a strict round robin fashion. I.e, after the first area has completed a read transfer, the second area should receive one transfer, and then the third, etc etc, and after all areas have been accessed once, the next block from the first area is read, and so on. The reason I want this type of access is to simulate a real time application that performs synchronous, sequential, reads from many files simultaneously. 

 

I have tried using many workers, with each worker having a different offset, however when I inspect the throughput for individual workers, they are not identical. If it were working the way I want it to, the throughput would be absolutely identical for every worker.

 

If IOMeter can't do what I want, I'd appreciate suggestions for alternatives.

 

Thanks,

Greg.


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Simulate sequential and strictly round-robing reads from many files simultaneously?

Greg Sullivan-3
RE: the round-robin reads, I've found a program that will do what I
want - fio (Flexible IO Tester). Since I don't want any disk queuing
at all, I configure it to do synchronous reads. (I'm on Windows, and
there's no synchronous engine in fio for Windows, however I was
advised that there is a workaround - use direct io and if writing, set
sync=1. It's working fine).

Greg.
p.s Matt - sorry - I accidentally made a couple of replies directly to you.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Simulate sequential and strictly round-robing reads from many files simultaneously?

Neto, Antonio Jose Rodrigues
Greg

Could you please post the fio configuration file

Thank you

neto

-----Original Message-----
From: Greg Sullivan [mailto:[hidden email]]
Sent: Wednesday, August 15, 2012 8:56 AM
To: Matt Alsip
Cc: [hidden email]
Subject: Re: [Iometer-user] Simulate sequential and strictly round-robing reads from many files simultaneously?

RE: the round-robin reads, I've found a program that will do what I want - fio (Flexible IO Tester). Since I don't want any disk queuing at all, I configure it to do synchronous reads. (I'm on Windows, and there's no synchronous engine in fio for Windows, however I was advised that there is a workaround - use direct io and if writing, set sync=1. It's working fine).

Greg.
p.s Matt - sorry - I accidentally made a couple of replies directly to you.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Simulate sequential and strictly round-robing reads from many files simultaneously?

Greg Sullivan-3
Neto - here is an example config:
>>> SNIP
[global]
; Sequentially read in a round-robin fashion from 10 files.
; Round-robin is the default method of io distribution, so no need to
specify it.
; On Unix, for synchronous io, specify ioengine=sync (for example)

sync=1 ; Required on Windows for synchronous writes.
direct=1 ; Required on Windows for synchronous io. Also bypasses file
system cache.

nrfiles=10 ; Will create the files if they don't already exist.

[read]
rw=read  ; Sequential reads
>>> SNIP

On 15 August 2012 23:25, Neto, Antonio Jose Rodrigues
<[hidden email]> wrote:

> Greg
>
> Could you please post the fio configuration file
>
> Thank you
>
> neto
>
> -----Original Message-----
> From: Greg Sullivan [mailto:[hidden email]]
> Sent: Wednesday, August 15, 2012 8:56 AM
> To: Matt Alsip
> Cc: [hidden email]
> Subject: Re: [Iometer-user] Simulate sequential and strictly round-robing reads from many files simultaneously?
>
> RE: the round-robin reads, I've found a program that will do what I want - fio (Flexible IO Tester). Since I don't want any disk queuing at all, I configure it to do synchronous reads. (I'm on Windows, and there's no synchronous engine in fio for Windows, however I was advised that there is a workaround - use direct io and if writing, set sync=1. It's working fine).
>
> Greg.
> p.s Matt - sorry - I accidentally made a couple of replies directly to you.
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Iometer-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/iometer-user

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Simulate sequential and strictly round-robing reads from many files simultaneously?

Greg Sullivan-3
Note that for strict, single-threaded operation, the io must be done
by one job. If seperate fio jobs do the fio, each job will be a
seperate process, and whilst each process will ensure that they issue
io with no queuing, there will still be queuing at the device, because
the processes operate independently. (this seems to be how IOMeter
works as well - there is no "global" queue depth setting in IOMeter,
as far as I can see).

The idea of accessing more than one file concurrently from a single
job in fio isn't documented properly I don't think. Also, if I allow
the io to be asynchronous, there isn't as much queuing (at the device)
as there is if each file has a seperate process. This is probably to
be expected, but it's not documented IMHO. I'm probably going to
comment about this on the fio mailing list.

Greg.


On 15 August 2012 23:36, Greg Sullivan <[hidden email]> wrote:

>
> Neto - here is an example config:
> >>> SNIP
> [global]
> ; Sequentially read in a round-robin fashion from 10 files.
> ; Round-robin is the default method of io distribution, so no need to
> specify it.
> ; On Unix, for synchronous io, specify ioengine=sync (for example)
>
> sync=1 ; Required on Windows for synchronous writes.
> direct=1 ; Required on Windows for synchronous io. Also bypasses file
> system cache.
>
> nrfiles=10 ; Will create the files if they don't already exist.
>
> [read]
> rw=read  ; Sequential reads
> >>> SNIP
>
> On 15 August 2012 23:25, Neto, Antonio Jose Rodrigues
> <[hidden email]> wrote:
> > Greg
> >
> > Could you please post the fio configuration file
> >
> > Thank you
> >
> > neto

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Loading...