Dynamo gets stuck at the end of testing

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Dynamo gets stuck at the end of testing

Bopp, Eric

Hello,

 

I’m looking for help with a problem I’m having running IoMeter.  The problem is that dynamo seems to hang when the first test completes.  Here is the output:

 

Command line parameter(s):

   Looking for Iometer on "10.221.220.20"

   New manager name is "taylor"

 

Sending login request...

   taylor

   10.221.220.49 (port 38153)

Successful PortTCP::Connect

  - port name: 10.221.220.20

 

*** If dynamo and iometer hangs here, please make sure

*** you use a correct -m <manager_computer_name> that

*** can ping from iometer machine. use IP if need.

   Login accepted.

Reporting drive information...

Set_Sizes: Open on "/sys/iobw.tst" failed (error No such file or directory).

Set_Sizes: Open on "/proc/sys/fs/binfmt_misc/iobw.tst" failed (error Permission denied).

  Physical drives (raw devices)...

Reporting TCP network information...

   done.

Removing worker 0.

Removing worker 0.

Worker 0 running Access Spec: 512B 0% reads 0% random

Access specifications for test defined.

Grunt: Growing grunt data buffer from 512 to 512

Worker 0 running Access Spec: 512B 0% reads 0% random

Access specifications for test defined.

Grunt: Grunt data buffer size 512 >= 512, returning

Worker 0 setting targets...

Seeding random Number Generator(1933595241135)

   Generating random data...

Starting...

   Opening targets.

Beginning to perform I/O...

   Stopped.

Stopping...

*** HANGS HERE ***

 

 

At this stage the IoMeter GUI is also frozen.

 

Here is the configuration I am using: 

 

IoMeter version 1.1.0-rc1 (iometer-1.1.0-rc1-src.tar.bz2)

IoMeter GUI is running on Windows XP. 

Dynamo is running on a linux box. Built dynamo in the linux test environment.  (make –f Makefile-linux.i386 all)

 

Thanks,

 

Eric

 

 


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Dynamo gets stuck at the end of testing

Neto, Antonio Jose Rodrigues
Hi Eric

This is neto from Brazil

How are you?

Please try to change the block size to 4096 bytes and do the test.

Thank you

All the best

neto

From: <Bopp>, Eric <[hidden email]>
Date: Tuesday, July 9, 2013 8:57 PM
To: "[hidden email]" <[hidden email]>
Subject: [Iometer-user] Dynamo gets stuck at the end of testing

Hello,

 

I’m looking for help with a problem I’m having running IoMeter.  The problem is that dynamo seems to hang when the first test completes.  Here is the output:

 

Command line parameter(s):

   Looking for Iometer on "10.221.220.20"

   New manager name is "taylor"

 

Sending login request...

   taylor

   10.221.220.49 (port 38153)

Successful PortTCP::Connect

  - port name: 10.221.220.20

 

*** If dynamo and iometer hangs here, please make sure

*** you use a correct -m <manager_computer_name> that

*** can ping from iometer machine. use IP if need.

   Login accepted.

Reporting drive information...

Set_Sizes: Open on "/sys/iobw.tst" failed (error No such file or directory).

Set_Sizes: Open on "/proc/sys/fs/binfmt_misc/iobw.tst" failed (error Permission denied).

  Physical drives (raw devices)...

Reporting TCP network information...

   done.

Removing worker 0.

Removing worker 0.

Worker 0 running Access Spec: 512B 0% reads 0% random

Access specifications for test defined.

Grunt: Growing grunt data buffer from 512 to 512

Worker 0 running Access Spec: 512B 0% reads 0% random

Access specifications for test defined.

Grunt: Grunt data buffer size 512 >= 512, returning

Worker 0 setting targets...

Seeding random Number Generator(1933595241135)

   Generating random data...

Starting...

   Opening targets.

Beginning to perform I/O...

   Stopped.

Stopping...

*** HANGS HERE ***

 

 

At this stage the IoMeter GUI is also frozen.

 

Here is the configuration I am using: 

 

IoMeter version 1.1.0-rc1 (iometer-1.1.0-rc1-src.tar.bz2)

IoMeter GUI is running on Windows XP. 

Dynamo is running on a linux box. Built dynamo in the linux test environment.  (make –f Makefile-linux.i386 all)

 

Thanks,

 

Eric

 

 


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Dynamo gets stuck at the end of testing

Bopp, Eric

Hi Neto,

 

I’m using the Default access specification.  I changed the transfer request size to 4 KB, but dynamo is still hanging when the test stops.

 

Thanks,

 

Eric

 

 

Additional info:  when the tests are running, the only results that get updated on the Results Display tab is CPU Utilization.  All of the other results are 0.

 

I rebuilt dynamo with  _DEBUG and _DETAILS=1.  Here is the output:

 

in while loop : Manager::Run()

Received 8 of 8 bytes from socket Taylor.

in Process_Message() : RECORD_OFF

   Getting system performance data.

   Getting system performance data.

   Getting system performance data.

CPU 0 recorded stat 0 = 2.86243

CPU 0 recorded stat 1 = 1.785

CPU 0 recorded stat 2 = 1.07743

CPU 0 recorded stat 3 = 0

CPU 0 recorded stat 4 = 0

CPU 0 recorded stat 5 = 2712.75

CPU 1 recorded stat 0 = 2.97097

CPU 1 recorded stat 1 = 1.27844

CPU 1 recorded stat 2 = 1.69253

CPU 1 recorded stat 3 = 0

CPU 1 recorded stat 4 = 0

CPU 1 recorded stat 5 = 0

TCP recorded stat 0 = 0

   Network interface 0 recorded stat 0 = 80.0273

   Network interface 0 recorded stat 1 = 0

   Network interface 0 recorded stat 2 = 0

   Network interface 1 recorded stat 0 = 657.823

   Network interface 1 recorded stat 1 = 0

   Network interface 1 recorded stat 2 = 0

   Network interface 2 recorded stat 0 = 0

   Network interface 2 recorded stat 1 = 0

   Network interface 2 recorded stat 2 = 0

   Getting system performance data.

   Getting system performance data.

   Getting system performance data.

CPU 0 recorded stat 0 = 2.86243

CPU 0 recorded stat 1 = 1.785

CPU 0 recorded stat 2 = 1.07743

CPU 0 recorded stat 3 = 0

CPU 0 recorded stat 4 = 0

CPU 0 recorded stat 5 = 2712.75

CPU 1 recorded stat 0 = 2.97097

CPU 1 recorded stat 1 = 1.27844

CPU 1 recorded stat 2 = 1.69253

CPU 1 recorded stat 3 = 0

CPU 1 recorded stat 4 = 0

CPU 1 recorded stat 5 = 0

TCP recorded stat 0 = 0

   Network interface 0 recorded stat 0 = 80.0273

   Network interface 0 recorded stat 1 = 0

   Network interface 0 recorded stat 2 = 0

   Network interface 1 recorded stat 0 = 657.819

   Network interface 1 recorded stat 1 = 0

   Network interface 1 recorded stat 2 = 0

   Network interface 2 recorded stat 0 = 0

   Network interface 2 recorded stat 1 = 0

   Network interface 2 recorded stat 2 = 0

   Stopped.

Recording stopped.

in SynchSend

Sent 8 of 8 bytes to socket Taylor.

in while loop : Manager::Run()

Received 8 of 8 bytes from socket Taylor.

in Process_Message() : STOP

Stopping...

*** HANGS HERE ***

 


From: Neto, Antonio Jose Rodrigues [mailto:[hidden email]]
Sent: Tuesday, July 09, 2013 6:49 PM
To: Bopp, Eric; [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

Hi Eric

 

This is neto from Brazil

 

How are you?

 

Please try to change the block size to 4096 bytes and do the test.

 

Thank you

 

All the best

 

neto

 

From: <Bopp>, Eric <[hidden email]>
Date: Tuesday, July 9, 2013 8:57 PM
To: "[hidden email]" <[hidden email]>
Subject: [Iometer-user] Dynamo gets stuck at the end of testing

 

Hello,

 

I’m looking for help with a problem I’m having running IoMeter.  The problem is that dynamo seems to hang when the first test completes.  Here is the output:

 

Command line parameter(s):

   Looking for Iometer on "10.221.220.20"

   New manager name is "taylor"

 

Sending login request...

   taylor

   10.221.220.49 (port 38153)

Successful PortTCP::Connect

  - port name: 10.221.220.20

 

*** If dynamo and iometer hangs here, please make sure

*** you use a correct -m <manager_computer_name> that

*** can ping from iometer machine. use IP if need.

   Login accepted.

Reporting drive information...

Set_Sizes: Open on "/sys/iobw.tst" failed (error No such file or directory).

Set_Sizes: Open on "/proc/sys/fs/binfmt_misc/iobw.tst" failed (error Permission denied).

  Physical drives (raw devices)...

Reporting TCP network information...

   done.

Removing worker 0.

Removing worker 0.

Worker 0 running Access Spec: 512B 0% reads 0% random

Access specifications for test defined.

Grunt: Growing grunt data buffer from 512 to 512

Worker 0 running Access Spec: 512B 0% reads 0% random

Access specifications for test defined.

Grunt: Grunt data buffer size 512 >= 512, returning

Worker 0 setting targets...

Seeding random Number Generator(1933595241135)

   Generating random data...

Starting...

   Opening targets.

Beginning to perform I/O...

   Stopped.

Stopping...

*** HANGS HERE ***

 

 

At this stage the IoMeter GUI is also frozen.

 

Here is the configuration I am using: 

 

IoMeter version 1.1.0-rc1 (iometer-1.1.0-rc1-src.tar.bz2)

IoMeter GUI is running on Windows XP. 

Dynamo is running on a linux box. Built dynamo in the linux test environment.  (make –f Makefile-linux.i386 all)

 

Thanks,

 

Eric

 

 


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Dynamo gets stuck at the end of testing

Chris Horneck
How about a gstack dump of the dynamo process when it's hung?

gstack <dynamo pid>

On Wed, Jul 10, 2013 at 4:11 PM, Bopp, Eric <[hidden email]> wrote:

Hi Neto,

 

I’m using the Default access specification.  I changed the transfer request size to 4 KB, but dynamo is still hanging when the test stops.

 

Thanks,

 

Eric

 

 

Additional info:  when the tests are running, the only results that get updated on the Results Display tab is CPU Utilization.  All of the other results are 0.

 

I rebuilt dynamo with  _DEBUG and _DETAILS=1.  Here is the output:

 

in while loop : Manager::Run()

Received 8 of 8 bytes from socket Taylor.

in Process_Message() : RECORD_OFF

   Getting system performance data.

   Getting system performance data.

   Getting system performance data.

CPU 0 recorded stat 0 = 2.86243

CPU 0 recorded stat 1 = 1.785

CPU 0 recorded stat 2 = 1.07743

CPU 0 recorded stat 3 = 0

CPU 0 recorded stat 4 = 0

CPU 0 recorded stat 5 = 2712.75

CPU 1 recorded stat 0 = 2.97097

CPU 1 recorded stat 1 = 1.27844

CPU 1 recorded stat 2 = 1.69253

CPU 1 recorded stat 3 = 0

CPU 1 recorded stat 4 = 0

CPU 1 recorded stat 5 = 0

TCP recorded stat 0 = 0

   Network interface 0 recorded stat 0 = 80.0273

   Network interface 0 recorded stat 1 = 0

   Network interface 0 recorded stat 2 = 0

   Network interface 1 recorded stat 0 = 657.823

   Network interface 1 recorded stat 1 = 0

   Network interface 1 recorded stat 2 = 0

   Network interface 2 recorded stat 0 = 0

   Network interface 2 recorded stat 1 = 0

   Network interface 2 recorded stat 2 = 0

   Getting system performance data.

   Getting system performance data.

   Getting system performance data.

CPU 0 recorded stat 0 = 2.86243

CPU 0 recorded stat 1 = 1.785

CPU 0 recorded stat 2 = 1.07743

CPU 0 recorded stat 3 = 0

CPU 0 recorded stat 4 = 0

CPU 0 recorded stat 5 = 2712.75

CPU 1 recorded stat 0 = 2.97097

CPU 1 recorded stat 1 = 1.27844

CPU 1 recorded stat 2 = 1.69253

CPU 1 recorded stat 3 = 0

CPU 1 recorded stat 4 = 0

CPU 1 recorded stat 5 = 0

TCP recorded stat 0 = 0

   Network interface 0 recorded stat 0 = 80.0273

   Network interface 0 recorded stat 1 = 0

   Network interface 0 recorded stat 2 = 0

   Network interface 1 recorded stat 0 = 657.819

   Network interface 1 recorded stat 1 = 0

   Network interface 1 recorded stat 2 = 0

   Network interface 2 recorded stat 0 = 0

   Network interface 2 recorded stat 1 = 0

   Network interface 2 recorded stat 2 = 0

   Stopped.

Recording stopped.

in SynchSend

Sent 8 of 8 bytes to socket Taylor.

in while loop : Manager::Run()

Received 8 of 8 bytes from socket Taylor.

in Process_Message() : STOP

Stopping...

*** HANGS HERE ***

 


From: Neto, Antonio Jose Rodrigues [mailto:[hidden email]]
Sent: Tuesday, July 09, 2013 6:49 PM
To: Bopp, Eric; [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

Hi Eric

 

This is neto from Brazil

 

How are you?

 

Please try to change the block size to 4096 bytes and do the test.

 

Thank you

 

All the best

 

neto

 

From: <Bopp>, Eric <[hidden email]>
Date: Tuesday, July 9, 2013 8:57 PM
To: "[hidden email]" <[hidden email]>
Subject: [Iometer-user] Dynamo gets stuck at the end of testing

 

Hello,

 

I’m looking for help with a problem I’m having running IoMeter.  The problem is that dynamo seems to hang when the first test completes.  Here is the output:

 

Command line parameter(s):

   Looking for Iometer on "10.221.220.20"

   New manager name is "taylor"

 

Sending login request...

   taylor

   10.221.220.49 (port 38153)

Successful PortTCP::Connect

  - port name: 10.221.220.20

 

*** If dynamo and iometer hangs here, please make sure

*** you use a correct -m <manager_computer_name> that

*** can ping from iometer machine. use IP if need.

   Login accepted.

Reporting drive information...

Set_Sizes: Open on "/sys/iobw.tst" failed (error No such file or directory).

Set_Sizes: Open on "/proc/sys/fs/binfmt_misc/iobw.tst" failed (error Permission denied).

  Physical drives (raw devices)...

Reporting TCP network information...

   done.

Removing worker 0.

Removing worker 0.

Worker 0 running Access Spec: 512B 0% reads 0% random

Access specifications for test defined.

Grunt: Growing grunt data buffer from 512 to 512

Worker 0 running Access Spec: 512B 0% reads 0% random

Access specifications for test defined.

Grunt: Grunt data buffer size 512 >= 512, returning

Worker 0 setting targets...

Seeding random Number Generator(1933595241135)

   Generating random data...

Starting...

   Opening targets.

Beginning to perform I/O...

   Stopped.

Stopping...

*** HANGS HERE ***

 

 

At this stage the IoMeter GUI is also frozen.

 

Here is the configuration I am using: 

 

IoMeter version 1.1.0-rc1 (iometer-1.1.0-rc1-src.tar.bz2)

IoMeter GUI is running on Windows XP. 

Dynamo is running on a linux box. Built dynamo in the linux test environment.  (make –f Makefile-linux.i386 all)

 

Thanks,

 

Eric

 

 


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user



------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Dynamo gets stuck at the end of testing

Vedran Degoricija
Hi Eric,
 
 
As Chris points out, getting a stack trace would certainly be useful. From the looks of it, your code may be waiting on the not_ready variable in Grunt::Wait_For_Stop(). But it's hard to envision the cause of it.
 
Regards,
Ved

From: Chris Horneck <[hidden email]>
To: "Bopp, Eric" <[hidden email]>
Cc: [hidden email]
Sent: Wednesday, July 10, 2013 2:51 PM
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

How about a gstack dump of the dynamo process when it's hung?

gstack <dynamo pid>

On Wed, Jul 10, 2013 at 4:11 PM, Bopp, Eric <[hidden email]> wrote:
Hi Neto,
 
I’m using the Default access specification.  I changed the transfer request size to 4 KB, but dynamo is still hanging when the test stops.
 
Thanks,
 
Eric
 
 
Additional info:  when the tests are running, the only results that get updated on the Results Display tab is CPU Utilization.  All of the other results are 0.
 
I rebuilt dynamo with  _DEBUG and _DETAILS=1.  Here is the output:
 
in while loop : Manager::Run()
Received 8 of 8 bytes from socket Taylor.
in Process_Message() : RECORD_OFF
   Getting system performance data.
   Getting system performance data.
   Getting system performance data.
CPU 0 recorded stat 0 = 2.86243
CPU 0 recorded stat 1 = 1.785
CPU 0 recorded stat 2 = 1.07743
CPU 0 recorded stat 3 = 0
CPU 0 recorded stat 4 = 0
CPU 0 recorded stat 5 = 2712.75
CPU 1 recorded stat 0 = 2.97097
CPU 1 recorded stat 1 = 1.27844
CPU 1 recorded stat 2 = 1.69253
CPU 1 recorded stat 3 = 0
CPU 1 recorded stat 4 = 0
CPU 1 recorded stat 5 = 0
TCP recorded stat 0 = 0
   Network interface 0 recorded stat 0 = 80.0273
   Network interface 0 recorded stat 1 = 0
   Network interface 0 recorded stat 2 = 0
   Network interface 1 recorded stat 0 = 657.823
   Network interface 1 recorded stat 1 = 0
   Network interface 1 recorded stat 2 = 0
   Network interface 2 recorded stat 0 = 0
   Network interface 2 recorded stat 1 = 0
   Network interface 2 recorded stat 2 = 0
   Getting system performance data.
   Getting system performance data.
   Getting system performance data.
CPU 0 recorded stat 0 = 2.86243
CPU 0 recorded stat 1 = 1.785
CPU 0 recorded stat 2 = 1.07743
CPU 0 recorded stat 3 = 0
CPU 0 recorded stat 4 = 0
CPU 0 recorded stat 5 = 2712.75
CPU 1 recorded stat 0 = 2.97097
CPU 1 recorded stat 1 = 1.27844
CPU 1 recorded stat 2 = 1.69253
CPU 1 recorded stat 3 = 0
CPU 1 recorded stat 4 = 0
CPU 1 recorded stat 5 = 0
TCP recorded stat 0 = 0
   Network interface 0 recorded stat 0 = 80.0273
   Network interface 0 recorded stat 1 = 0
   Network interface 0 recorded stat 2 = 0
   Network interface 1 recorded stat 0 = 657.819
   Network interface 1 recorded stat 1 = 0
   Network interface 1 recorded stat 2 = 0
   Network interface 2 recorded stat 0 = 0
   Network interface 2 recorded stat 1 = 0
   Network interface 2 recorded stat 2 = 0
   Stopped.
Recording stopped.
in SynchSend
Sent 8 of 8 bytes to socket Taylor.
in while loop : Manager::Run()
Received 8 of 8 bytes from socket Taylor.
in Process_Message() : STOP
Stopping...
*** HANGS HERE ***
 
From: Neto, Antonio Jose Rodrigues [mailto:[hidden email]]
Sent: Tuesday, July 09, 2013 6:49 PM
To: Bopp, Eric; [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing
 
Hi Eric
 
This is neto from Brazil
 
How are you?
 
Please try to change the block size to 4096 bytes and do the test.
 
Thank you
 
All the best
 
neto
 
From: <Bopp>, Eric <[hidden email]>
Date: Tuesday, July 9, 2013 8:57 PM
To: "[hidden email]" <[hidden email]>
Subject: [Iometer-user] Dynamo gets stuck at the end of testing
 
Hello,
 
I’m looking for help with a problem I’m having running IoMeter.  The problem is that dynamo seems to hang when the first test completes.  Here is the output:
 
Command line parameter(s):
   Looking for Iometer on "10.221.220.20"
   New manager name is "taylor"
 
Sending login request...
   taylor
   10.221.220.49 (port 38153)
Successful PortTCP::Connect
  - port name: 10.221.220.20
 
*** If dynamo and iometer hangs here, please make sure
*** you use a correct -m <manager_computer_name> that
*** can ping from iometer machine. use IP if need.
   Login accepted.
Reporting drive information...
Set_Sizes: Open on "/sys/iobw.tst" failed (error No such file or directory).
Set_Sizes: Open on "/proc/sys/fs/binfmt_misc/iobw.tst" failed (error Permission denied).
  Physical drives (raw devices)...
Reporting TCP network information...
   done.
Removing worker 0.
Removing worker 0.
Worker 0 running Access Spec: 512B 0% reads 0% random
Access specifications for test defined.
Grunt: Growing grunt data buffer from 512 to 512
Worker 0 running Access Spec: 512B 0% reads 0% random
Access specifications for test defined.
Grunt: Grunt data buffer size 512 >= 512, returning
Worker 0 setting targets...
Seeding random Number Generator(1933595241135)
   Generating random data...
Starting...
   Opening targets.
Beginning to perform I/O...
   Stopped.
Stopping...
*** HANGS HERE ***
 
 
At this stage the IoMeter GUI is also frozen.
 
Here is the configuration I am using: 
 
IoMeter version 1.1.0-rc1 (iometer-1.1.0-rc1-src.tar.bz2)
IoMeter GUI is running on Windows XP. 
Dynamo is running on a linux box. Built dynamo in the linux test environment.  (make –f Makefile-linux.i386 all)
 
Thanks,
 
Eric
 
 

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________

------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user



------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user



------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Dynamo gets stuck at the end of testing

Bopp, Eric

There are 2 threads running when it gets stuck (see backtraces below).  Since I don’t see any results other than cpu utilization, I’m wondering whether the disk test is actually running (maybe that’s why there is a problem shutting it down?).  Can anyone tell from the detailed debug in the previous message whether the test is actually running?

 

I think it is probably a problem with my setup, but I haven’t been able to figure out what it is.

 

 

(gdb) info threads

  2 Thread 0xb727fbc0 (LWP 17944)  0xffffe424 in __kernel_vsyscall ()

* 1 Thread 0xb749a6c0 (LWP 17352)  0xffffe424 in __kernel_vsyscall ()

(gdb) thread 2

[Switching to thread 2 (Thread 0xb727fbc0 (LWP 17944))]#0  0xffffe424 in __kernel_vsyscall ()

(gdb) bt

#0  0xffffe424 in __kernel_vsyscall ()

#1  0xb76e552e in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0

#2  0xb76e334b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0

#3  0xb76f11ba in aio_suspend64 () from /lib/tls/librt.so.1

#4  0x0805c752 in GetQueuedCompletionStatus ()

#5  0x0805beae in CQAIO::GetStatus ()

#6  0x08051512 in Grunt::Complete_IO ()

#7  0x08051b2c in Grunt::Do_IOs ()

#8  0x0805245a in Grunt_Thread_Wrapper ()

#9  0xb76e1073 in start_thread () from /lib/tls/libpthread.so.0

#10 0xb756d73e in clone () from /lib/tls/libc.so.6

(gdb) thread 1

[Switching to thread 1 (Thread 0xb749a6c0 (LWP 17352))]#0  0xffffe424 in __kernel_vsyscall ()

(gdb) bt

#0  0xffffe424 in __kernel_vsyscall ()

#1  0xb752c0f6 in nanosleep () from /lib/tls/libc.so.6

#2  0xb75663da in usleep () from /lib/tls/libc.so.6

#3  0x08050509 in Grunt::Wait_For_Stop ()

#4  0x0804c9b4 in Manager::Stop_Test ()

#5  0x0804e9c7 in Manager::Process_Message ()

#6  0x0804eba5 in Manager::Run ()

#7  0x0804bf81 in main ()

 

 

Thanks,

 

Eric

 


From: Vedran Degoricija [mailto:[hidden email]]
Sent: Thursday, July 11, 2013 12:30 AM
To: Chris Horneck; Bopp, Eric
Cc: [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

Hi Eric,

 

 

As Chris points out, getting a stack trace would certainly be useful. From the looks of it, your code may be waiting on the not_ready variable in Grunt::Wait_For_Stop(). But it's hard to envision the cause of it.

 

Regards,

Ved

 

From: Chris Horneck <[hidden email]>
To: "Bopp, Eric" <[hidden email]>
Cc: [hidden email]
Sent: Wednesday, July 10, 2013 2:51 PM
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

How about a gstack dump of the dynamo process when it's hung?

 

gstack <dynamo pid>

On Wed, Jul 10, 2013 at 4:11 PM, Bopp, Eric <[hidden email]> wrote:

Hi Neto,

 

I’m using the Default access specification.  I changed the transfer request size to 4 KB, but dynamo is still hanging when the test stops.

 

Thanks,

 

Eric

 

 

Additional info:  when the tests are running, the only results that get updated on the Results Display tab is CPU Utilization.  All of the other results are 0.

 

I rebuilt dynamo with  _DEBUG and _DETAILS=1.  Here is the output:

 

in while loop : Manager::Run()

Received 8 of 8 bytes from socket Taylor.

in Process_Message() : RECORD_OFF

   Getting system performance data.

   Getting system performance data.

   Getting system performance data.

CPU 0 recorded stat 0 = 2.86243

CPU 0 recorded stat 1 = 1.785

CPU 0 recorded stat 2 = 1.07743

CPU 0 recorded stat 3 = 0

CPU 0 recorded stat 4 = 0

CPU 0 recorded stat 5 = 2712.75

CPU 1 recorded stat 0 = 2.97097

CPU 1 recorded stat 1 = 1.27844

CPU 1 recorded stat 2 = 1.69253

CPU 1 recorded stat 3 = 0

CPU 1 recorded stat 4 = 0

CPU 1 recorded stat 5 = 0

TCP recorded stat 0 = 0

   Network interface 0 recorded stat 0 = 80.0273

   Network interface 0 recorded stat 1 = 0

   Network interface 0 recorded stat 2 = 0

   Network interface 1 recorded stat 0 = 657.823

   Network interface 1 recorded stat 1 = 0

   Network interface 1 recorded stat 2 = 0

   Network interface 2 recorded stat 0 = 0

   Network interface 2 recorded stat 1 = 0

   Network interface 2 recorded stat 2 = 0

   Getting system performance data.

   Getting system performance data.

   Getting system performance data.

CPU 0 recorded stat 0 = 2.86243

CPU 0 recorded stat 1 = 1.785

CPU 0 recorded stat 2 = 1.07743

CPU 0 recorded stat 3 = 0

CPU 0 recorded stat 4 = 0

CPU 0 recorded stat 5 = 2712.75

CPU 1 recorded stat 0 = 2.97097

CPU 1 recorded stat 1 = 1.27844

CPU 1 recorded stat 2 = 1.69253

CPU 1 recorded stat 3 = 0

CPU 1 recorded stat 4 = 0

CPU 1 recorded stat 5 = 0

TCP recorded stat 0 = 0

   Network interface 0 recorded stat 0 = 80.0273

   Network interface 0 recorded stat 1 = 0

   Network interface 0 recorded stat 2 = 0

   Network interface 1 recorded stat 0 = 657.819

   Network interface 1 recorded stat 1 = 0

   Network interface 1 recorded stat 2 = 0

   Network interface 2 recorded stat 0 = 0

   Network interface 2 recorded stat 1 = 0

   Network interface 2 recorded stat 2 = 0

   Stopped.

Recording stopped.

in SynchSend

Sent 8 of 8 bytes to socket Taylor.

in while loop : Manager::Run()

Received 8 of 8 bytes from socket Taylor.

in Process_Message() : STOP

Stopping...

*** HANGS HERE ***

 

From: Neto, Antonio Jose Rodrigues [mailto:[hidden email]]
Sent: Tuesday, July 09, 2013 6:49 PM
To: Bopp, Eric; [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

Hi Eric

 

This is neto from Brazil

 

How are you?

 

Please try to change the block size to 4096 bytes and do the test.

 

Thank you

 

All the best

 

neto

 

From: <Bopp>, Eric <[hidden email]>
Date: Tuesday, July 9, 2013 8:57 PM
To: "[hidden email]" <[hidden email]>
Subject: [Iometer-user] Dynamo gets stuck at the end of testing

 

Hello,

 

I’m looking for help with a problem I’m having running IoMeter.  The problem is that dynamo seems to hang when the first test completes.  Here is the output:

 

Command line parameter(s):

   Looking for Iometer on "10.221.220.20"

   New manager name is "taylor"

 

Sending login request...

   taylor

   10.221.220.49 (port 38153)

Successful PortTCP::Connect

  - port name: 10.221.220.20

 

*** If dynamo and iometer hangs here, please make sure

*** you use a correct -m <manager_computer_name> that

*** can ping from iometer machine. use IP if need.

   Login accepted.

Reporting drive information...

Set_Sizes: Open on "/sys/iobw.tst" failed (error No such file or directory).

Set_Sizes: Open on "/proc/sys/fs/binfmt_misc/iobw.tst" failed (error Permission denied).

  Physical drives (raw devices)...

Reporting TCP network information...

   done.

Removing worker 0.

Removing worker 0.

Worker 0 running Access Spec: 512B 0% reads 0% random

Access specifications for test defined.

Grunt: Growing grunt data buffer from 512 to 512

Worker 0 running Access Spec: 512B 0% reads 0% random

Access specifications for test defined.

Grunt: Grunt data buffer size 512 >= 512, returning

Worker 0 setting targets...

Seeding random Number Generator(1933595241135)

   Generating random data...

Starting...

   Opening targets.

Beginning to perform I/O...

   Stopped.

Stopping...

*** HANGS HERE ***

 

 

At this stage the IoMeter GUI is also frozen.

 

Here is the configuration I am using: 

 

IoMeter version 1.1.0-rc1 (iometer-1.1.0-rc1-src.tar.bz2)

IoMeter GUI is running on Windows XP. 

Dynamo is running on a linux box. Built dynamo in the linux test environment.  (make –f Makefile-linux.i386 all)

 

Thanks,

 

Eric

 

 


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user

 


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Dynamo gets stuck at the end of testing

Vedran Degoricija
Hi Eric,
 
I don't think there is any IO going on. The reason for the high CPU utilization is that one of the threads (thread 1) is spinning in the Grunt::Wait_For_Stop() waiting for a variable to update, and it never does. The reason for this appears in the other thread (thread 2) where the call to aio_suspend64 is hung.  I don't know enough about the Linux kernel to be certain if this is AIO related or controlled by another setting.
 
You could try rebuilding dynamo with IOMTR_SETTING_LINUX_LIBAIO removed from the makefile and see what happens.
 
Regards,
Ved
 
From: "Bopp, Eric" <[hidden email]>
To: Vedran Degoricija <[hidden email]>; Chris Horneck <[hidden email]>
Cc: [hidden email]
Sent: Thursday, July 11, 2013 8:20 AM
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

There are 2 threads running when it gets stuck (see backtraces below).  Since I don’t see any results other than cpu utilization, I’m wondering whether the disk test is actually running (maybe that’s why there is a problem shutting it down?).  Can anyone tell from the detailed debug in the previous message whether the test is actually running?
 
I think it is probably a problem with my setup, but I haven’t been able to figure out what it is.
 
 
(gdb) info threads
  2 Thread 0xb727fbc0 (LWP 17944)  0xffffe424 in __kernel_vsyscall ()
* 1 Thread 0xb749a6c0 (LWP 17352)  0xffffe424 in __kernel_vsyscall ()
(gdb) thread 2
[Switching to thread 2 (Thread 0xb727fbc0 (LWP 17944))]#0  0xffffe424 in __kernel_vsyscall ()
(gdb) bt
#0  0xffffe424 in __kernel_vsyscall ()
#1  0xb76e552e in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0
#2  0xb76e334b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
#3  0xb76f11ba in aio_suspend64 () from /lib/tls/librt.so.1
#4  0x0805c752 in GetQueuedCompletionStatus ()
#5  0x0805beae in CQAIO::GetStatus ()
#6  0x08051512 in Grunt::Complete_IO ()
#7  0x08051b2c in Grunt::Do_IOs ()
#8  0x0805245a in Grunt_Thread_Wrapper ()
#9  0xb76e1073 in start_thread () from /lib/tls/libpthread.so.0
#10 0xb756d73e in clone () from /lib/tls/libc.so.6
(gdb) thread 1
[Switching to thread 1 (Thread 0xb749a6c0 (LWP 17352))]#0  0xffffe424 in __kernel_vsyscall ()
(gdb) bt
#0  0xffffe424 in __kernel_vsyscall ()
#1  0xb752c0f6 in nanosleep () from /lib/tls/libc.so.6
#2  0xb75663da in usleep () from /lib/tls/libc.so.6
#3  0x08050509 in Grunt::Wait_For_Stop ()
#4  0x0804c9b4 in Manager::Stop_Test ()
#5  0x0804e9c7 in Manager::Process_Message ()
#6  0x0804eba5 in Manager::Run ()
#7  0x0804bf81 in main ()
 
 
Thanks,
 
Eric
 
From: Vedran Degoricija [mailto:[hidden email]]
Sent: Thursday, July 11, 2013 12:30 AM
To: Chris Horneck; Bopp, Eric
Cc: [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing
 
Hi Eric,
 
 
As Chris points out, getting a stack trace would certainly be useful. From the looks of it, your code may be waiting on the not_ready variable in Grunt::Wait_For_Stop(). But it's hard to envision the cause of it.
 
Regards,
Ved
 
From: Chris Horneck <[hidden email]>
To: "Bopp, Eric" <[hidden email]>
Cc: [hidden email]
Sent: Wednesday, July 10, 2013 2:51 PM
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing
 
How about a gstack dump of the dynamo process when it's hung?
 
gstack <dynamo pid>
On Wed, Jul 10, 2013 at 4:11 PM, Bopp, Eric <[hidden email]> wrote:
Hi Neto,
 
I’m using the Default access specification.  I changed the transfer request size to 4 KB, but dynamo is still hanging when the test stops.
 
Thanks,
 
Eric
 
 
Additional info:  when the tests are running, the only results that get updated on the Results Display tab is CPU Utilization.  All of the other results are 0.
 
I rebuilt dynamo with  _DEBUG and _DETAILS=1.  Here is the output:
 
in while loop : Manager::Run()
Received 8 of 8 bytes from socket Taylor .
in Process_Message() : RECORD_OFF
   Getting system performance data.
   Getting system performance data.
   Getting system performance data.
CPU 0 recorded stat 0 = 2.86243
CPU 0 recorded stat 1 = 1.785
CPU 0 recorded stat 2 = 1.07743
CPU 0 recorded stat 3 = 0
CPU 0 recorded stat 4 = 0
CPU 0 recorded stat 5 = 2712.75
CPU 1 recorded stat 0 = 2.97097
CPU 1 recorded stat 1 = 1.27844
CPU 1 recorded stat 2 = 1.69253
CPU 1 recorded stat 3 = 0
CPU 1 recorded stat 4 = 0
CPU 1 recorded stat 5 = 0
TCP recorded stat 0 = 0
   Network interface 0 recorded stat 0 = 80.0273
   Network interface 0 recorded stat 1 = 0
   Network interface 0 recorded stat 2 = 0
   Network interface 1 recorded stat 0 = 657.823
   Network interface 1 recorded stat 1 = 0
   Network interface 1 recorded stat 2 = 0
   Network interface 2 recorded stat 0 = 0
   Network interface 2 recorded stat 1 = 0
   Network interface 2 recorded stat 2 = 0
   Getting system performance data.
   Getting system performance data.
   Getting system performance data.
CPU 0 recorded stat 0 = 2.86243
CPU 0 recorded stat 1 = 1.785
CPU 0 recorded stat 2 = 1.07743
CPU 0 recorded stat 3 = 0
CPU 0 recorded stat 4 = 0
CPU 0 recorded stat 5 = 2712.75
CPU 1 recorded stat 0 = 2.97097
CPU 1 recorded stat 1 = 1.27844
CPU 1 recorded stat 2 = 1.69253
CPU 1 recorded stat 3 = 0
CPU 1 recorded stat 4 = 0
CPU 1 recorded stat 5 = 0
TCP recorded stat 0 = 0
   Network interface 0 recorded stat 0 = 80.0273
   Network interface 0 recorded stat 1 = 0
   Network interface 0 recorded stat 2 = 0
   Network interface 1 recorded stat 0 = 657.819
   Network interface 1 recorded stat 1 = 0
   Network interface 1 recorded stat 2 = 0
   Network interface 2 recorded stat 0 = 0
   Network interface 2 recorded stat 1 = 0
   Network interface 2 recorded stat 2 = 0
   Stopped.
Recording stopped.
in SynchSend
Sent 8 of 8 bytes to socket Taylor .
in while loop : Manager::Run()
Received 8 of 8 bytes from socket Taylor .
in Process_Message() : STOP
Stopping...
*** HANGS HERE ***
 
From: Neto, Antonio Jose Rodrigues [mailto:[hidden email]]
Sent: Tuesday, July 09, 2013 6:49 PM
To: Bopp, Eric; [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing
 
Hi Eric
 
This is neto from Brazil
 
How are you?
 
Please try to change the block size to 4096 bytes and do the test.
 
Thank you
 
All the best
 
neto
 
From: <Bopp>, Eric <[hidden email]>
Date: Tuesday, July 9, 2013 8:57 PM
To: "[hidden email]" <[hidden email]>
Subject: [Iometer-user] Dynamo gets stuck at the end of testing
 
Hello,
 
I’m looking for help with a problem I’m having running IoMeter.  The problem is that dynamo seems to hang when the first test completes.  Here is the output:
 
Command line parameter(s):
   Looking for Iometer on "10.221.220.20"
   New manager name is " taylor "
 
Sending login request...
   taylor
   10.221.220.49 (port 38153)
Successful PortTCP::Connect
  - port name: 10.221.220.20
 
*** If dynamo and iometer hangs here, please make sure
*** you use a correct -m <manager_computer_name> that
*** can ping from iometer machine. use IP if need.
   Login accepted.
Reporting drive information...
Set_Sizes: Open on "/sys/iobw.tst" failed (error No such file or directory).
Set_Sizes: Open on "/proc/sys/fs/binfmt_misc/iobw.tst" failed (error Permission denied).
  Physical drives (raw devices)...
Reporting TCP network information...
   done.
Removing worker 0.
Removing worker 0.
Worker 0 running Access Spec: 512B 0% reads 0% random
Access specifications for test defined.
Grunt: Growing grunt data buffer from 512 to 512
Worker 0 running Access Spec: 512B 0% reads 0% random
Access specifications for test defined.
Grunt: Grunt data buffer size 512 >= 512, returning
Worker 0 setting targets...
Seeding random Number Generator(1933595241135)
   Generating random data...
Starting...
   Opening targets.
Beginning to perform I/O...
   Stopped.
Stopping...
*** HANGS HERE ***
 
 
At this stage the IoMeter GUI is also frozen.
 
Here is the configuration I am using: 
 
IoMeter version 1.1.0-rc1 (iometer-1.1.0-rc1-src.tar.bz2)
IoMeter GUI is running on Windows XP. 
Dynamo is running on a linux box. Built dynamo in the linux test environment.  (make –f Makefile-linux.i386 all)
 
Thanks,
 
Eric
 
 

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________

------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
 

------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________

------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user



------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Dynamo gets stuck at the end of testing

Bopp, Eric

I was originally using version 1.1.0-rc1.  That version doesn’t have the IOMTR_SETTING_LINUX_LIBAIO option.

 

I got the latest version (r141), which has IOMTR_SETTING_LINUX_LIBAIO off by default.  I saw the same issues when running this version.

 

I then built r141 with IOMTR_SETTING_LINUX_LIBAIO defined.  This version works a lot better:  I see results on the Results Display tab and the Stop button works.

 

Thanks to everyone who took a look at this issue.

 

Eric  

 


From: Vedran Degoricija [mailto:[hidden email]]
Sent: Thursday, July 11, 2013 11:01 PM
To: Bopp, Eric; Chris Horneck
Cc: [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

Hi Eric,

 

I don't think there is any IO going on. The reason for the high CPU utilization is that one of the threads (thread 1) is spinning in the Grunt::Wait_For_Stop() waiting for a variable to update, and it never does. The reason for this appears in the other thread (thread 2) where the call to aio_suspend64 is hung.  I don't know enough about the Linux kernel to be certain if this is AIO related or controlled by another setting.

 

You could try rebuilding dynamo with IOMTR_SETTING_LINUX_LIBAIO removed from the makefile and see what happens.

 

Regards,

Ved

 

From: "Bopp, Eric" <[hidden email]>
To: Vedran Degoricija <[hidden email]>; Chris Horneck <[hidden email]>
Cc: [hidden email]
Sent: Thursday, July 11, 2013 8:20 AM
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

There are 2 threads running when it gets stuck (see backtraces below).  Since I don’t see any results other than cpu utilization, I’m wondering whether the disk test is actually running (maybe that’s why there is a problem shutting it down?).  Can anyone tell from the detailed debug in the previous message whether the test is actually running?

 

I think it is probably a problem with my setup, but I haven’t been able to figure out what it is.

 

 

(gdb) info threads

  2 Thread 0xb727fbc0 (LWP 17944)  0xffffe424 in __kernel_vsyscall ()

* 1 Thread 0xb749a6c0 (LWP 17352)  0xffffe424 in __kernel_vsyscall ()

(gdb) thread 2

[Switching to thread 2 (Thread 0xb727fbc0 (LWP 17944))]#0  0xffffe424 in __kernel_vsyscall ()

(gdb) bt

#0  0xffffe424 in __kernel_vsyscall ()

#1  0xb76e552e in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0

#2  0xb76e334b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0

#3  0xb76f11ba in aio_suspend64 () from /lib/tls/librt.so.1

#4  0x0805c752 in GetQueuedCompletionStatus ()

#5  0x0805beae in CQAIO::GetStatus ()

#6  0x08051512 in Grunt::Complete_IO ()

#7  0x08051b2c in Grunt::Do_IOs ()

#8  0x0805245a in Grunt_Thread_Wrapper ()

#9  0xb76e1073 in start_thread () from /lib/tls/libpthread.so.0

#10 0xb756d73e in clone () from /lib/tls/libc.so.6

(gdb) thread 1

[Switching to thread 1 (Thread 0xb749a6c0 (LWP 17352))]#0  0xffffe424 in __kernel_vsyscall ()

(gdb) bt

#0  0xffffe424 in __kernel_vsyscall ()

#1  0xb752c0f6 in nanosleep () from /lib/tls/libc.so.6

#2  0xb75663da in usleep () from /lib/tls/libc.so.6

#3  0x08050509 in Grunt::Wait_For_Stop ()

#4  0x0804c9b4 in Manager::Stop_Test ()

#5  0x0804e9c7 in Manager::Process_Message ()

#6  0x0804eba5 in Manager::Run ()

#7  0x0804bf81 in main ()

 

 

Thanks,

 

Eric

 

From: Vedran Degoricija [mailto:[hidden email]]
Sent: Thursday, July 11, 2013 12:30 AM
To: Chris Horneck; Bopp, Eric
Cc: [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

Hi Eric,

 

 

As Chris points out, getting a stack trace would certainly be useful. From the looks of it, your code may be waiting on the not_ready variable in Grunt::Wait_For_Stop(). But it's hard to envision the cause of it.

 

Regards,

Ved

 

From: Chris Horneck <[hidden email]>
To: "Bopp, Eric" <[hidden email]>
Cc: [hidden email]
Sent: Wednesday, July 10, 2013 2:51 PM
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

How about a gstack dump of the dynamo process when it's hung?

 

gstack <dynamo pid>

On Wed, Jul 10, 2013 at 4:11 PM, Bopp, Eric <[hidden email]> wrote:

Hi Neto,

 

I’m using the Default access specification.  I changed the transfer request size to 4 KB, but dynamo is still hanging when the test stops.

 

Thanks,

 

Eric

 

 

Additional info:  when the tests are running, the only results that get updated on the Results Display tab is CPU Utilization.  All of the other results are 0.

 

I rebuilt dynamo with  _DEBUG and _DETAILS=1.  Here is the output:

 

in while loop : Manager::Run()

Received 8 of 8 bytes from socket Taylor .

in Process_Message() : RECORD_OFF

   Getting system performance data.

   Getting system performance data.

   Getting system performance data.

CPU 0 recorded stat 0 = 2.86243

CPU 0 recorded stat 1 = 1.785

CPU 0 recorded stat 2 = 1.07743

CPU 0 recorded stat 3 = 0

CPU 0 recorded stat 4 = 0

CPU 0 recorded stat 5 = 2712.75

CPU 1 recorded stat 0 = 2.97097

CPU 1 recorded stat 1 = 1.27844

CPU 1 recorded stat 2 = 1.69253

CPU 1 recorded stat 3 = 0

CPU 1 recorded stat 4 = 0

CPU 1 recorded stat 5 = 0

TCP recorded stat 0 = 0

   Network interface 0 recorded stat 0 = 80.0273

   Network interface 0 recorded stat 1 = 0

   Network interface 0 recorded stat 2 = 0

   Network interface 1 recorded stat 0 = 657.823

   Network interface 1 recorded stat 1 = 0

   Network interface 1 recorded stat 2 = 0

   Network interface 2 recorded stat 0 = 0

   Network interface 2 recorded stat 1 = 0

   Network interface 2 recorded stat 2 = 0

   Getting system performance data.

   Getting system performance data.

   Getting system performance data.

CPU 0 recorded stat 0 = 2.86243

CPU 0 recorded stat 1 = 1.785

CPU 0 recorded stat 2 = 1.07743

CPU 0 recorded stat 3 = 0

CPU 0 recorded stat 4 = 0

CPU 0 recorded stat 5 = 2712.75

CPU 1 recorded stat 0 = 2.97097

CPU 1 recorded stat 1 = 1.27844

CPU 1 recorded stat 2 = 1.69253

CPU 1 recorded stat 3 = 0

CPU 1 recorded stat 4 = 0

CPU 1 recorded stat 5 = 0

TCP recorded stat 0 = 0

   Network interface 0 recorded stat 0 = 80.0273

   Network interface 0 recorded stat 1 = 0

   Network interface 0 recorded stat 2 = 0

   Network interface 1 recorded stat 0 = 657.819

   Network interface 1 recorded stat 1 = 0

   Network interface 1 recorded stat 2 = 0

   Network interface 2 recorded stat 0 = 0

   Network interface 2 recorded stat 1 = 0

   Network interface 2 recorded stat 2 = 0

   Stopped.

Recording stopped.

in SynchSend

Sent 8 of 8 bytes to socket Taylor .

in while loop : Manager::Run()

Received 8 of 8 bytes from socket Taylor .

in Process_Message() : STOP

Stopping...

*** HANGS HERE ***

 

From: Neto, Antonio Jose Rodrigues [mailto:[hidden email]]
Sent: Tuesday, July 09, 2013 6:49 PM
To: Bopp, Eric; [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

Hi Eric

 

This is neto from Brazil

 

How are you?

 

Please try to change the block size to 4096 bytes and do the test.

 

Thank you

 

All the best

 

neto

 

From: <Bopp>, Eric <[hidden email]>
Date: Tuesday, July 9, 2013 8:57 PM
To: "[hidden email]" <[hidden email]>
Subject: [Iometer-user] Dynamo gets stuck at the end of testing

 

Hello,

 

I’m looking for help with a problem I’m having running IoMeter.  The problem is that dynamo seems to hang when the first test completes.  Here is the output:

 

Command line parameter(s):

   Looking for Iometer on "10.221.220.20"

   New manager name is " taylor "

 

Sending login request...

   taylor

   10.221.220.49 (port 38153)

Successful PortTCP::Connect

  - port name: 10.221.220.20

 

*** If dynamo and iometer hangs here, please make sure

*** you use a correct -m <manager_computer_name> that

*** can ping from iometer machine. use IP if need.

   Login accepted.

Reporting drive information...

Set_Sizes: Open on "/sys/iobw.tst" failed (error No such file or directory).

Set_Sizes: Open on "/proc/sys/fs/binfmt_misc/iobw.tst" failed (error Permission denied).

  Physical drives (raw devices)...

Reporting TCP network information...

   done.

Removing worker 0.

Removing worker 0.

Worker 0 running Access Spec: 512B 0% reads 0% random

Access specifications for test defined.

Grunt: Growing grunt data buffer from 512 to 512

Worker 0 running Access Spec: 512B 0% reads 0% random

Access specifications for test defined.

Grunt: Grunt data buffer size 512 >= 512, returning

Worker 0 setting targets...

Seeding random Number Generator(1933595241135)

   Generating random data...

Starting...

   Opening targets.

Beginning to perform I/O...

   Stopped.

Stopping...

*** HANGS HERE ***

 

 

At this stage the IoMeter GUI is also frozen.

 

Here is the configuration I am using: 

 

IoMeter version 1.1.0-rc1 (iometer-1.1.0-rc1-src.tar.bz2)

IoMeter GUI is running on Windows XP. 

Dynamo is running on a linux box. Built dynamo in the linux test environment.  (make –f Makefile-linux.i386 all)

 

Thanks,

 

Eric

 

 


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user

 


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Dynamo gets stuck at the end of testing

Gruher, Joseph R

Does this correct (or have other changes already corrected) the performance issues dynamo used to have under Linux?  For a long time dynamo had an issue where it would essentially run with no queue depth on Linux resulting in poor performance.  I believe this had something to do with the IO or thread type dynamo was using.  Sorry I can’t be more specific, I don’t really recall the details at this point.

 

Thanks,

Joe

 

From: Bopp, Eric [mailto:[hidden email]]
Sent: Tuesday, July 16, 2013 4:03 PM
To: [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

I was originally using version 1.1.0-rc1.  That version doesn’t have the IOMTR_SETTING_LINUX_LIBAIO option.

 

I got the latest version (r141), which has IOMTR_SETTING_LINUX_LIBAIO off by default.  I saw the same issues when running this version.

 

I then built r141 with IOMTR_SETTING_LINUX_LIBAIO defined.  This version works a lot better:  I see results on the Results Display tab and the Stop button works.

 

Thanks to everyone who took a look at this issue.

 

Eric  

 


From: Vedran Degoricija [[hidden email]]
Sent: Thursday, July 11, 2013 11:01 PM
To: Bopp, Eric; Chris Horneck
Cc: [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

Hi Eric,

 

I don't think there is any IO going on. The reason for the high CPU utilization is that one of the threads (thread 1) is spinning in the Grunt::Wait_For_Stop() waiting for a variable to update, and it never does. The reason for this appears in the other thread (thread 2) where the call to aio_suspend64 is hung.  I don't know enough about the Linux kernel to be certain if this is AIO related or controlled by another setting.

 

You could try rebuilding dynamo with IOMTR_SETTING_LINUX_LIBAIO removed from the makefile and see what happens.

 

Regards,

Ved

 

From: "Bopp, Eric" <[hidden email]>
To: Vedran Degoricija <[hidden email]>; Chris Horneck <[hidden email]>
Cc: [hidden email]
Sent: Thursday, July 11, 2013 8:20 AM
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

There are 2 threads running when it gets stuck (see backtraces below).  Since I don’t see any results other than cpu utilization, I’m wondering whether the disk test is actually running (maybe that’s why there is a problem shutting it down?).  Can anyone tell from the detailed debug in the previous message whether the test is actually running?

 

I think it is probably a problem with my setup, but I haven’t been able to figure out what it is.

 

 

(gdb) info threads

  2 Thread 0xb727fbc0 (LWP 17944)  0xffffe424 in __kernel_vsyscall ()

* 1 Thread 0xb749a6c0 (LWP 17352)  0xffffe424 in __kernel_vsyscall ()

(gdb) thread 2

[Switching to thread 2 (Thread 0xb727fbc0 (LWP 17944))]#0  0xffffe424 in __kernel_vsyscall ()

(gdb) bt

#0  0xffffe424 in __kernel_vsyscall ()

#1  0xb76e552e in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0

#2  0xb76e334b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0

#3  0xb76f11ba in aio_suspend64 () from /lib/tls/librt.so.1

#4  0x0805c752 in GetQueuedCompletionStatus ()

#5  0x0805beae in CQAIO::GetStatus ()

#6  0x08051512 in Grunt::Complete_IO ()

#7  0x08051b2c in Grunt::Do_IOs ()

#8  0x0805245a in Grunt_Thread_Wrapper ()

#9  0xb76e1073 in start_thread () from /lib/tls/libpthread.so.0

#10 0xb756d73e in clone () from /lib/tls/libc.so.6

(gdb) thread 1

[Switching to thread 1 (Thread 0xb749a6c0 (LWP 17352))]#0  0xffffe424 in __kernel_vsyscall ()

(gdb) bt

#0  0xffffe424 in __kernel_vsyscall ()

#1  0xb752c0f6 in nanosleep () from /lib/tls/libc.so.6

#2  0xb75663da in usleep () from /lib/tls/libc.so.6

#3  0x08050509 in Grunt::Wait_For_Stop ()

#4  0x0804c9b4 in Manager::Stop_Test ()

#5  0x0804e9c7 in Manager::Process_Message ()

#6  0x0804eba5 in Manager::Run ()

#7  0x0804bf81 in main ()

 

 

Thanks,

 

Eric

 

From: Vedran Degoricija [[hidden email]]
Sent: Thursday, July 11, 2013 12:30 AM
To: Chris Horneck; Bopp, Eric
Cc: [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

Hi Eric,

 

 

As Chris points out, getting a stack trace would certainly be useful. From the looks of it, your code may be waiting on the not_ready variable in Grunt::Wait_For_Stop(). But it's hard to envision the cause of it.

 

Regards,

Ved

 

From: Chris Horneck <[hidden email]>
To: "Bopp, Eric" <[hidden email]>
Cc: [hidden email]
Sent: Wednesday, July 10, 2013 2:51 PM
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

How about a gstack dump of the dynamo process when it's hung?

 

gstack <dynamo pid>

On Wed, Jul 10, 2013 at 4:11 PM, Bopp, Eric <[hidden email]> wrote:

Hi Neto,

 

I’m using the Default access specification.  I changed the transfer request size to 4 KB, but dynamo is still hanging when the test stops.

 

Thanks,

 

Eric

 

 

Additional info:  when the tests are running, the only results that get updated on the Results Display tab is CPU Utilization.  All of the other results are 0.

 

I rebuilt dynamo with  _DEBUG and _DETAILS=1.  Here is the output:

 

in while loop : Manager::Run()

Received 8 of 8 bytes from socket Taylor .

in Process_Message() : RECORD_OFF

   Getting system performance data.

   Getting system performance data.

   Getting system performance data.

CPU 0 recorded stat 0 = 2.86243

CPU 0 recorded stat 1 = 1.785

CPU 0 recorded stat 2 = 1.07743

CPU 0 recorded stat 3 = 0

CPU 0 recorded stat 4 = 0

CPU 0 recorded stat 5 = 2712.75

CPU 1 recorded stat 0 = 2.97097

CPU 1 recorded stat 1 = 1.27844

CPU 1 recorded stat 2 = 1.69253

CPU 1 recorded stat 3 = 0

CPU 1 recorded stat 4 = 0

CPU 1 recorded stat 5 = 0

TCP recorded stat 0 = 0

   Network interface 0 recorded stat 0 = 80.0273

   Network interface 0 recorded stat 1 = 0

   Network interface 0 recorded stat 2 = 0

   Network interface 1 recorded stat 0 = 657.823

   Network interface 1 recorded stat 1 = 0

   Network interface 1 recorded stat 2 = 0

   Network interface 2 recorded stat 0 = 0

   Network interface 2 recorded stat 1 = 0

   Network interface 2 recorded stat 2 = 0

   Getting system performance data.

   Getting system performance data.

   Getting system performance data.

CPU 0 recorded stat 0 = 2.86243

CPU 0 recorded stat 1 = 1.785

CPU 0 recorded stat 2 = 1.07743

CPU 0 recorded stat 3 = 0

CPU 0 recorded stat 4 = 0

CPU 0 recorded stat 5 = 2712.75

CPU 1 recorded stat 0 = 2.97097

CPU 1 recorded stat 1 = 1.27844

CPU 1 recorded stat 2 = 1.69253

CPU 1 recorded stat 3 = 0

CPU 1 recorded stat 4 = 0

CPU 1 recorded stat 5 = 0

TCP recorded stat 0 = 0

   Network interface 0 recorded stat 0 = 80.0273

   Network interface 0 recorded stat 1 = 0

   Network interface 0 recorded stat 2 = 0

   Network interface 1 recorded stat 0 = 657.819

   Network interface 1 recorded stat 1 = 0

   Network interface 1 recorded stat 2 = 0

   Network interface 2 recorded stat 0 = 0

   Network interface 2 recorded stat 1 = 0

   Network interface 2 recorded stat 2 = 0

   Stopped.

Recording stopped.

in SynchSend

Sent 8 of 8 bytes to socket Taylor .

in while loop : Manager::Run()

Received 8 of 8 bytes from socket Taylor .

in Process_Message() : STOP

Stopping...

*** HANGS HERE ***

 

From: Neto, Antonio Jose Rodrigues [mailto:[hidden email]]
Sent: Tuesday, July 09, 2013 6:49 PM
To: Bopp, Eric; [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

Hi Eric

 

This is neto from Brazil

 

How are you?

 

Please try to change the block size to 4096 bytes and do the test.

 

Thank you

 

All the best

 

neto

 

From: <Bopp>, Eric <[hidden email]>
Date: Tuesday, July 9, 2013 8:57 PM
To: "[hidden email]" <[hidden email]>
Subject: [Iometer-user] Dynamo gets stuck at the end of testing

 

Hello,

 

I’m looking for help with a problem I’m having running IoMeter.  The problem is that dynamo seems to hang when the first test completes.  Here is the output:

 

Command line parameter(s):

   Looking for Iometer on "10.221.220.20"

   New manager name is " taylor "

 

Sending login request...

   taylor

   10.221.220.49 (port 38153)

Successful PortTCP::Connect

  - port name: 10.221.220.20

 

*** If dynamo and iometer hangs here, please make sure

*** you use a correct -m <manager_computer_name> that

*** can ping from iometer machine. use IP if need.

   Login accepted.

Reporting drive information...

Set_Sizes: Open on "/sys/iobw.tst" failed (error No such file or directory).

Set_Sizes: Open on "/proc/sys/fs/binfmt_misc/iobw.tst" failed (error Permission denied).

  Physical drives (raw devices)...

Reporting TCP network information...

   done.

Removing worker 0.

Removing worker 0.

Worker 0 running Access Spec: 512B 0% reads 0% random

Access specifications for test defined.

Grunt: Growing grunt data buffer from 512 to 512

Worker 0 running Access Spec: 512B 0% reads 0% random

Access specifications for test defined.

Grunt: Grunt data buffer size 512 >= 512, returning

Worker 0 setting targets...

Seeding random Number Generator(1933595241135)

   Generating random data...

Starting...

   Opening targets.

Beginning to perform I/O...

   Stopped.

Stopping...

*** HANGS HERE ***

 

 

At this stage the IoMeter GUI is also frozen.

 

Here is the configuration I am using: 

 

IoMeter version 1.1.0-rc1 (iometer-1.1.0-rc1-src.tar.bz2)

IoMeter GUI is running on Windows XP. 

Dynamo is running on a linux box. Built dynamo in the linux test environment.  (make –f Makefile-linux.i386 all)

 

Thanks,

 

Eric

 

 


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user

 


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Dynamo gets stuck at the end of testing

Vedran Degoricija
Hi Joe,
 
Yes, the new code based on LIBAIO is supposed to do what you describe. :)
 
Ved

From: "Gruher, Joseph R" <[hidden email]>
To: "[hidden email]" <[hidden email]>
Cc: "Gruher, Joseph R" <[hidden email]>
Sent: Tuesday, July 16, 2013 4:11 PM
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

Does this correct (or have other changes already corrected) the performance issues dynamo used to have under Linux?  For a long time dynamo had an issue where it would essentially run with no queue depth on Linux resulting in poor performance.  I believe this had something to do with the IO or thread type dynamo was using.  Sorry I can’t be more specific, I don’t really recall the details at this point.
 
Thanks,
Joe
 
From: Bopp, Eric [mailto:[hidden email]]
Sent: Tuesday, July 16, 2013 4:03 PM
To: [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing
 
I was originally using version 1.1.0-rc1.  That version doesn’t have the IOMTR_SETTING_LINUX_LIBAIO option.
 
I got the latest version (r141), which has IOMTR_SETTING_LINUX_LIBAIO off by default.  I saw the same issues when running this version.
 
I then built r141 with IOMTR_SETTING_LINUX_LIBAIO defined.  This version works a lot better:  I see results on the Results Display tab and the Stop button works.
 
Thanks to everyone who took a look at this issue.
 
Eric  
 
From: Vedran Degoricija [[hidden email]]
Sent: Thursday, July 11, 2013 11:01 PM
To: Bopp, Eric; Chris Horneck
Cc: [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing
 
Hi Eric,
 
I don't think there is any IO going on. The reason for the high CPU utilization is that one of the threads (thread 1) is spinning in the Grunt::Wait_For_Stop() waiting for a variable to update, and it never does. The reason for this appears in the other thread (thread 2) where the call to aio_suspend64 is hung.  I don't know enough about the Linux kernel to be certain if this is AIO related or controlled by another setting.
 
You could try rebuilding dynamo with IOMTR_SETTING_LINUX_LIBAIO removed from the makefile and see what happens.
 
Regards,
Ved
 
From: "Bopp, Eric" <[hidden email]>
To: Vedran Degoricija <[hidden email]>; Chris Horneck <[hidden email]>
Cc: [hidden email]
Sent: Thursday, July 11, 2013 8:20 AM
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing
 
There are 2 threads running when it gets stuck (see backtraces below).  Since I don’t see any results other than cpu utilization, I’m wondering whether the disk test is actually running (maybe that’s why there is a problem shutting it down?).  Can anyone tell from the detailed debug in the previous message whether the test is actually running?
 
I think it is probably a problem with my setup, but I haven’t been able to figure out what it is.
 
 
(gdb) info threads
  2 Thread 0xb727fbc0 (LWP 17944)  0xffffe424 in __kernel_vsyscall ()
* 1 Thread 0xb749a6c0 (LWP 17352)  0xffffe424 in __kernel_vsyscall ()
(gdb) thread 2
[Switching to thread 2 (Thread 0xb727fbc0 (LWP 17944))]#0  0xffffe424 in __kernel_vsyscall ()
(gdb) bt
#0  0xffffe424 in __kernel_vsyscall ()
#1  0xb76e552e in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0
#2  0xb76e334b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
#3  0xb76f11ba in aio_suspend64 () from /lib/tls/librt.so.1
#4  0x0805c752 in GetQueuedCompletionStatus ()
#5  0x0805beae in CQAIO::GetStatus ()
#6  0x08051512 in Grunt::Complete_IO ()
#7  0x08051b2c in Grunt::Do_IOs ()
#8  0x0805245a in Grunt_Thread_Wrapper ()
#9  0xb76e1073 in start_thread () from /lib/tls/libpthread.so.0
#10 0xb756d73e in clone () from /lib/tls/libc.so.6
(gdb) thread 1
[Switching to thread 1 (Thread 0xb749a6c0 (LWP 17352))]#0  0xffffe424 in __kernel_vsyscall ()
(gdb) bt
#0  0xffffe424 in __kernel_vsyscall ()
#1  0xb752c0f6 in nanosleep () from /lib/tls/libc.so.6
#2  0xb75663da in usleep () from /lib/tls/libc.so.6
#3  0x08050509 in Grunt::Wait_For_Stop ()
#4  0x0804c9b4 in Manager::Stop_Test ()
#5  0x0804e9c7 in Manager::Process_Message ()
#6  0x0804eba5 in Manager::Run ()
#7  0x0804bf81 in main ()
 
 
Thanks,
 
Eric
 
From: Vedran Degoricija [[hidden email]]
Sent: Thursday, July 11, 2013 12:30 AM
To: Chris Horneck; Bopp, Eric
Cc: [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing
 
Hi Eric,
 
 
As Chris points out, getting a stack trace would certainly be useful. From the looks of it, your code may be waiting on the not_ready variable in Grunt::Wait_For_Stop(). But it's hard to envision the cause of it.
 
Regards,
Ved
 
From: Chris Horneck <[hidden email]>
To: "Bopp, Eric" <[hidden email]>
Cc: [hidden email]
Sent: Wednesday, July 10, 2013 2:51 PM
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing
 
How about a gstack dump of the dynamo process when it's hung?
 
gstack <dynamo pid>
On Wed, Jul 10, 2013 at 4:11 PM, Bopp, Eric <[hidden email]> wrote:
Hi Neto,
 
I’m using the Default access specification.  I changed the transfer request size to 4 KB, but dynamo is still hanging when the test stops.
 
Thanks,
 
Eric
 
 
Additional info:  when the tests are running, the only results that get updated on the Results Display tab is CPU Utilization.  All of the other results are 0.
 
I rebuilt dynamo with  _DEBUG and _DETAILS=1.  Here is the output:
 
in while loop : Manager::Run()
Received 8 of 8 bytes from socket Taylor .
in Process_Message() : RECORD_OFF
   Getting system performance data.
   Getting system performance data.
   Getting system performance data.
CPU 0 recorded stat 0 = 2.86243
CPU 0 recorded stat 1 = 1.785
CPU 0 recorded stat 2 = 1.07743
CPU 0 recorded stat 3 = 0
CPU 0 recorded stat 4 = 0
CPU 0 recorded stat 5 = 2712.75
CPU 1 recorded stat 0 = 2.97097
CPU 1 recorded stat 1 = 1.27844
CPU 1 recorded stat 2 = 1.69253
CPU 1 recorded stat 3 = 0
CPU 1 recorded stat 4 = 0
CPU 1 recorded stat 5 = 0
TCP recorded stat 0 = 0
   Network interface 0 recorded stat 0 = 80.0273
   Network interface 0 recorded stat 1 = 0
   Network interface 0 recorded stat 2 = 0
   Network interface 1 recorded stat 0 = 657.823
   Network interface 1 recorded stat 1 = 0
   Network interface 1 recorded stat 2 = 0
   Network interface 2 recorded stat 0 = 0
   Network interface 2 recorded stat 1 = 0
   Network interface 2 recorded stat 2 = 0
   Getting system performance data.
   Getting system performance data.
   Getting system performance data.
CPU 0 recorded stat 0 = 2.86243
CPU 0 recorded stat 1 = 1.785
CPU 0 recorded stat 2 = 1.07743
CPU 0 recorded stat 3 = 0
CPU 0 recorded stat 4 = 0
CPU 0 recorded stat 5 = 2712.75
CPU 1 recorded stat 0 = 2.97097
CPU 1 recorded stat 1 = 1.27844
CPU 1 recorded stat 2 = 1.69253
CPU 1 recorded stat 3 = 0
CPU 1 recorded stat 4 = 0
CPU 1 recorded stat 5 = 0
TCP recorded stat 0 = 0
   Network interface 0 recorded stat 0 = 80.0273
   Network interface 0 recorded stat 1 = 0
   Network interface 0 recorded stat 2 = 0
   Network interface 1 recorded stat 0 = 657.819
   Network interface 1 recorded stat 1 = 0
   Network interface 1 recorded stat 2 = 0
   Network interface 2 recorded stat 0 = 0
   Network interface 2 recorded stat 1 = 0
   Network interface 2 recorded stat 2 = 0
   Stopped.
Recording stopped.
in SynchSend
Sent 8 of 8 bytes to socket Taylor .
in while loop : Manager::Run()
Received 8 of 8 bytes from socket Taylor .
in Process_Message() : STOP
Stopping...
*** HANGS HERE ***
 
From: Neto, Antonio Jose Rodrigues [mailto:[hidden email]]
Sent: Tuesday, July 09, 2013 6:49 PM
To: Bopp, Eric; [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing
 
Hi Eric
 
This is neto from Brazil
 
How are you?
 
Please try to change the block size to 4096 bytes and do the test.
 
Thank you
 
All the best
 
neto
 
From: <Bopp>, Eric <[hidden email]>
Date: Tuesday, July 9, 2013 8:57 PM
To: "[hidden email]" <[hidden email]>
Subject: [Iometer-user] Dynamo gets stuck at the end of testing
 
Hello,
 
I’m looking for help with a problem I’m having running IoMeter.  The problem is that dynamo seems to hang when the first test completes.  Here is the output:
 
Command line parameter(s):
   Looking for Iometer on "10.221.220.20"
   New manager name is " taylor "
 
Sending login request...
   taylor
   10.221.220.49 (port 38153)
Successful PortTCP::Connect
  - port name: 10.221.220.20
 
*** If dynamo and iometer hangs here, please make sure
*** you use a correct -m <manager_computer_name> that
*** can ping from iometer machine. use IP if need.
   Login accepted.
Reporting drive information...
Set_Sizes: Open on "/sys/iobw.tst" failed (error No such file or directory).
Set_Sizes: Open on "/proc/sys/fs/binfmt_misc/iobw.tst" failed (error Permission denied).
  Physical drives (raw devices)...
Reporting TCP network information...
   done.
Removing worker 0.
Removing worker 0.
Worker 0 running Access Spec: 512B 0% reads 0% random
Access specifications for test defined.
Grunt: Growing grunt data buffer from 512 to 512
Worker 0 running Access Spec: 512B 0% reads 0% random
Access specifications for test defined.
Grunt: Grunt data buffer size 512 >= 512, returning
Worker 0 setting targets...
Seeding random Number Generator(1933595241135)
   Generating random data...
Starting...
   Opening targets.
Beginning to perform I/O...
   Stopped.
Stopping...
*** HANGS HERE ***
 
 
At this stage the IoMeter GUI is also frozen.
 
Here is the configuration I am using: 
 
IoMeter version 1.1.0-rc1 (iometer-1.1.0-rc1-src.tar.bz2)
IoMeter GUI is running on Windows XP. 
Dynamo is running on a linux box. Built dynamo in the linux test environment.  (make –f Makefile-linux.i386 all)
 
Thanks,
 
Eric
 
 

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________

------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
 

------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________

------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________

------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user



------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Dynamo gets stuck at the end of testing

Gruher, Joseph R

Where can I grab r141 (or whatever is the latest)?  On the SourceForge page, even under the development file tree, I only see 1.1.0-rc1 as the latest available.

 

Thanks,

Joe

 

From: Vedran Degoricija [mailto:[hidden email]]
Sent: Wednesday, July 17, 2013 11:43 PM
To: Gruher, Joseph R; [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

Hi Joe,

 

Yes, the new code based on LIBAIO is supposed to do what you describe. :)

 

Ved

 

From: "Gruher, Joseph R" <[hidden email]>
To: "[hidden email]" <[hidden email]>
Cc: "Gruher, Joseph R" <[hidden email]>
Sent: Tuesday, July 16, 2013 4:11 PM
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

Does this correct (or have other changes already corrected) the performance issues dynamo used to have under Linux?  For a long time dynamo had an issue where it would essentially run with no queue depth on Linux resulting in poor performance.  I believe this had something to do with the IO or thread type dynamo was using.  Sorry I can’t be more specific, I don’t really recall the details at this point.

 

Thanks,

Joe

 

From: Bopp, Eric [[hidden email]]
Sent: Tuesday, July 16, 2013 4:03 PM
To: [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

I was originally using version 1.1.0-rc1.  That version doesn’t have the IOMTR_SETTING_LINUX_LIBAIO option.

 

I got the latest version (r141), which has IOMTR_SETTING_LINUX_LIBAIO off by default.  I saw the same issues when running this version.

 

I then built r141 with IOMTR_SETTING_LINUX_LIBAIO defined.  This version works a lot better:  I see results on the Results Display tab and the Stop button works.

 

Thanks to everyone who took a look at this issue.

 

Eric  

 

From: Vedran Degoricija [[hidden email]]
Sent: Thursday, July 11, 2013 11:01 PM
To: Bopp, Eric; Chris Horneck
Cc: [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

Hi Eric,

 

I don't think there is any IO going on. The reason for the high CPU utilization is that one of the threads (thread 1) is spinning in the Grunt::Wait_For_Stop() waiting for a variable to update, and it never does. The reason for this appears in the other thread (thread 2) where the call to aio_suspend64 is hung.  I don't know enough about the Linux kernel to be certain if this is AIO related or controlled by another setting.

 

You could try rebuilding dynamo with IOMTR_SETTING_LINUX_LIBAIO removed from the makefile and see what happens.

 

Regards,

Ved

 

From: "Bopp, Eric" <[hidden email]>
To: Vedran Degoricija <[hidden email]>; Chris Horneck <[hidden email]>
Cc: [hidden email]
Sent: Thursday, July 11, 2013 8:20 AM
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

There are 2 threads running when it gets stuck (see backtraces below).  Since I don’t see any results other than cpu utilization, I’m wondering whether the disk test is actually running (maybe that’s why there is a problem shutting it down?).  Can anyone tell from the detailed debug in the previous message whether the test is actually running?

 

I think it is probably a problem with my setup, but I haven’t been able to figure out what it is.

 

 

(gdb) info threads

  2 Thread 0xb727fbc0 (LWP 17944)  0xffffe424 in __kernel_vsyscall ()

* 1 Thread 0xb749a6c0 (LWP 17352)  0xffffe424 in __kernel_vsyscall ()

(gdb) thread 2

[Switching to thread 2 (Thread 0xb727fbc0 (LWP 17944))]#0  0xffffe424 in __kernel_vsyscall ()

(gdb) bt

#0  0xffffe424 in __kernel_vsyscall ()

#1  0xb76e552e in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0

#2  0xb76e334b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0

#3  0xb76f11ba in aio_suspend64 () from /lib/tls/librt.so.1

#4  0x0805c752 in GetQueuedCompletionStatus ()

#5  0x0805beae in CQAIO::GetStatus ()

#6  0x08051512 in Grunt::Complete_IO ()

#7  0x08051b2c in Grunt::Do_IOs ()

#8  0x0805245a in Grunt_Thread_Wrapper ()

#9  0xb76e1073 in start_thread () from /lib/tls/libpthread.so.0

#10 0xb756d73e in clone () from /lib/tls/libc.so.6

(gdb) thread 1

[Switching to thread 1 (Thread 0xb749a6c0 (LWP 17352))]#0  0xffffe424 in __kernel_vsyscall ()

(gdb) bt

#0  0xffffe424 in __kernel_vsyscall ()

#1  0xb752c0f6 in nanosleep () from /lib/tls/libc.so.6

#2  0xb75663da in usleep () from /lib/tls/libc.so.6

#3  0x08050509 in Grunt::Wait_For_Stop ()

#4  0x0804c9b4 in Manager::Stop_Test ()

#5  0x0804e9c7 in Manager::Process_Message ()

#6  0x0804eba5 in Manager::Run ()

#7  0x0804bf81 in main ()

 

 

Thanks,

 

Eric

 

From: Vedran Degoricija [[hidden email]]
Sent: Thursday, July 11, 2013 12:30 AM
To: Chris Horneck; Bopp, Eric
Cc: [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

Hi Eric,

 

 

As Chris points out, getting a stack trace would certainly be useful. From the looks of it, your code may be waiting on the not_ready variable in Grunt::Wait_For_Stop(). But it's hard to envision the cause of it.

 

Regards,

Ved

 

From: Chris Horneck <[hidden email]>
To: "Bopp, Eric" <[hidden email]>
Cc: [hidden email]
Sent: Wednesday, July 10, 2013 2:51 PM
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

How about a gstack dump of the dynamo process when it's hung?

 

gstack <dynamo pid>

On Wed, Jul 10, 2013 at 4:11 PM, Bopp, Eric <[hidden email]> wrote:

Hi Neto,

 

I’m using the Default access specification.  I changed the transfer request size to 4 KB, but dynamo is still hanging when the test stops.

 

Thanks,

 

Eric

 

 

Additional info:  when the tests are running, the only results that get updated on the Results Display tab is CPU Utilization.  All of the other results are 0.

 

I rebuilt dynamo with  _DEBUG and _DETAILS=1.  Here is the output:

 

in while loop : Manager::Run()

Received 8 of 8 bytes from socket Taylor .

in Process_Message() : RECORD_OFF

   Getting system performance data.

   Getting system performance data.

   Getting system performance data.

CPU 0 recorded stat 0 = 2.86243

CPU 0 recorded stat 1 = 1.785

CPU 0 recorded stat 2 = 1.07743

CPU 0 recorded stat 3 = 0

CPU 0 recorded stat 4 = 0

CPU 0 recorded stat 5 = 2712.75

CPU 1 recorded stat 0 = 2.97097

CPU 1 recorded stat 1 = 1.27844

CPU 1 recorded stat 2 = 1.69253

CPU 1 recorded stat 3 = 0

CPU 1 recorded stat 4 = 0

CPU 1 recorded stat 5 = 0

TCP recorded stat 0 = 0

   Network interface 0 recorded stat 0 = 80.0273

   Network interface 0 recorded stat 1 = 0

   Network interface 0 recorded stat 2 = 0

   Network interface 1 recorded stat 0 = 657.823

   Network interface 1 recorded stat 1 = 0

   Network interface 1 recorded stat 2 = 0

   Network interface 2 recorded stat 0 = 0

   Network interface 2 recorded stat 1 = 0

   Network interface 2 recorded stat 2 = 0

   Getting system performance data.

   Getting system performance data.

   Getting system performance data.

CPU 0 recorded stat 0 = 2.86243

CPU 0 recorded stat 1 = 1.785

CPU 0 recorded stat 2 = 1.07743

CPU 0 recorded stat 3 = 0

CPU 0 recorded stat 4 = 0

CPU 0 recorded stat 5 = 2712.75

CPU 1 recorded stat 0 = 2.97097

CPU 1 recorded stat 1 = 1.27844

CPU 1 recorded stat 2 = 1.69253

CPU 1 recorded stat 3 = 0

CPU 1 recorded stat 4 = 0

CPU 1 recorded stat 5 = 0

TCP recorded stat 0 = 0

   Network interface 0 recorded stat 0 = 80.0273

   Network interface 0 recorded stat 1 = 0

   Network interface 0 recorded stat 2 = 0

   Network interface 1 recorded stat 0 = 657.819

   Network interface 1 recorded stat 1 = 0

   Network interface 1 recorded stat 2 = 0

   Network interface 2 recorded stat 0 = 0

   Network interface 2 recorded stat 1 = 0

   Network interface 2 recorded stat 2 = 0

   Stopped.

Recording stopped.

in SynchSend

Sent 8 of 8 bytes to socket Taylor .

in while loop : Manager::Run()

Received 8 of 8 bytes from socket Taylor .

in Process_Message() : STOP

Stopping...

*** HANGS HERE ***

 

From: Neto, Antonio Jose Rodrigues [mailto:[hidden email]]
Sent: Tuesday, July 09, 2013 6:49 PM
To: Bopp, Eric; [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

Hi Eric

 

This is neto from Brazil

 

How are you?

 

Please try to change the block size to 4096 bytes and do the test.

 

Thank you

 

All the best

 

neto

 

From: <Bopp>, Eric <[hidden email]>
Date: Tuesday, July 9, 2013 8:57 PM
To: "[hidden email]" <[hidden email]>
Subject: [Iometer-user] Dynamo gets stuck at the end of testing

 

Hello,

 

I’m looking for help with a problem I’m having running IoMeter.  The problem is that dynamo seems to hang when the first test completes.  Here is the output:

 

Command line parameter(s):

   Looking for Iometer on "10.221.220.20"

   New manager name is " taylor "

 

Sending login request...

   taylor

   10.221.220.49 (port 38153)

Successful PortTCP::Connect

  - port name: 10.221.220.20

 

*** If dynamo and iometer hangs here, please make sure

*** you use a correct -m <manager_computer_name> that

*** can ping from iometer machine. use IP if need.

   Login accepted.

Reporting drive information...

Set_Sizes: Open on "/sys/iobw.tst" failed (error No such file or directory).

Set_Sizes: Open on "/proc/sys/fs/binfmt_misc/iobw.tst" failed (error Permission denied).

  Physical drives (raw devices)...

Reporting TCP network information...

   done.

Removing worker 0.

Removing worker 0.

Worker 0 running Access Spec: 512B 0% reads 0% random

Access specifications for test defined.

Grunt: Growing grunt data buffer from 512 to 512

Worker 0 running Access Spec: 512B 0% reads 0% random

Access specifications for test defined.

Grunt: Grunt data buffer size 512 >= 512, returning

Worker 0 setting targets...

Seeding random Number Generator(1933595241135)

   Generating random data...

Starting...

   Opening targets.

Beginning to perform I/O...

   Stopped.

Stopping...

*** HANGS HERE ***

 

 

At this stage the IoMeter GUI is also frozen.

 

Here is the configuration I am using: 

 

IoMeter version 1.1.0-rc1 (iometer-1.1.0-rc1-src.tar.bz2)

IoMeter GUI is running on Windows XP. 

Dynamo is running on a linux box. Built dynamo in the linux test environment.  (make –f Makefile-linux.i386 all)

 

Thanks,

 

Eric

 

 


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user

 


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user


------------------------------------------------------------------------------
Get your SQL database under version control now!
Version control is standard for application code, but databases havent
caught up. So what steps can you take to put your SQL databases under
version control? Why should you start doing it? Read more to find out.
http://pubads.g.doubleclick.net/gampad/clk?id=49501711&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Dynamo gets stuck at the end of testing

juriggs
unsubscribe


On Tue, Jul 30, 2013 at 11:46 AM, Gruher, Joseph R <[hidden email]> wrote:

Where can I grab r141 (or whatever is the latest)?  On the SourceForge page, even under the development file tree, I only see 1.1.0-rc1 as the latest available.

 

Thanks,

Joe

 

From: Vedran Degoricija [mailto:[hidden email]]
Sent: Wednesday, July 17, 2013 11:43 PM
To: Gruher, Joseph R; [hidden email]


Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

Hi Joe,

 

Yes, the new code based on LIBAIO is supposed to do what you describe. :)

 

Ved

 

From: "Gruher, Joseph R" <[hidden email]>
To: "[hidden email]" <[hidden email]>
Cc: "Gruher, Joseph R" <[hidden email]>
Sent: Tuesday, July 16, 2013 4:11 PM
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

Does this correct (or have other changes already corrected) the performance issues dynamo used to have under Linux?  For a long time dynamo had an issue where it would essentially run with no queue depth on Linux resulting in poor performance.  I believe this had something to do with the IO or thread type dynamo was using.  Sorry I can’t be more specific, I don’t really recall the details at this point.

 

Thanks,

Joe

 

From: Bopp, Eric [[hidden email]]
Sent: Tuesday, July 16, 2013 4:03 PM
To: [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

I was originally using version 1.1.0-rc1.  That version doesn’t have the IOMTR_SETTING_LINUX_LIBAIO option.

 

I got the latest version (r141), which has IOMTR_SETTING_LINUX_LIBAIO off by default.  I saw the same issues when running this version.

 

I then built r141 with IOMTR_SETTING_LINUX_LIBAIO defined.  This version works a lot better:  I see results on the Results Display tab and the Stop button works.

 

Thanks to everyone who took a look at this issue.

 

Eric  

 

From: Vedran Degoricija [[hidden email]]
Sent: Thursday, July 11, 2013 11:01 PM
To: Bopp, Eric; Chris Horneck
Cc: [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

Hi Eric,

 

I don't think there is any IO going on. The reason for the high CPU utilization is that one of the threads (thread 1) is spinning in the Grunt::Wait_For_Stop() waiting for a variable to update, and it never does. The reason for this appears in the other thread (thread 2) where the call to aio_suspend64 is hung.  I don't know enough about the Linux kernel to be certain if this is AIO related or controlled by another setting.

 

You could try rebuilding dynamo with IOMTR_SETTING_LINUX_LIBAIO removed from the makefile and see what happens.

 

Regards,

Ved

 

From: "Bopp, Eric" <[hidden email]>
To: Vedran Degoricija <[hidden email]>; Chris Horneck <[hidden email]>
Cc: [hidden email]
Sent: Thursday, July 11, 2013 8:20 AM
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

There are 2 threads running when it gets stuck (see backtraces below).  Since I don’t see any results other than cpu utilization, I’m wondering whether the disk test is actually running (maybe that’s why there is a problem shutting it down?).  Can anyone tell from the detailed debug in the previous message whether the test is actually running?

 

I think it is probably a problem with my setup, but I haven’t been able to figure out what it is.

 

 

(gdb) info threads

  2 Thread 0xb727fbc0 (LWP 17944)  0xffffe424 in __kernel_vsyscall ()

* 1 Thread 0xb749a6c0 (LWP 17352)  0xffffe424 in __kernel_vsyscall ()

(gdb) thread 2

[Switching to thread 2 (Thread 0xb727fbc0 (LWP 17944))]#0  0xffffe424 in __kernel_vsyscall ()

(gdb) bt

#0  0xffffe424 in __kernel_vsyscall ()

#1  0xb76e552e in __lll_mutex_lock_wait () from /lib/tls/libpthread.so.0

#2  0xb76e334b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0

#3  0xb76f11ba in aio_suspend64 () from /lib/tls/librt.so.1

#4  0x0805c752 in GetQueuedCompletionStatus ()

#5  0x0805beae in CQAIO::GetStatus ()

#6  0x08051512 in Grunt::Complete_IO ()

#7  0x08051b2c in Grunt::Do_IOs ()

#8  0x0805245a in Grunt_Thread_Wrapper ()

#9  0xb76e1073 in start_thread () from /lib/tls/libpthread.so.0

#10 0xb756d73e in clone () from /lib/tls/libc.so.6

(gdb) thread 1

[Switching to thread 1 (Thread 0xb749a6c0 (LWP 17352))]#0  0xffffe424 in __kernel_vsyscall ()

(gdb) bt

#0  0xffffe424 in __kernel_vsyscall ()

#1  0xb752c0f6 in nanosleep () from /lib/tls/libc.so.6

#2  0xb75663da in usleep () from /lib/tls/libc.so.6

#3  0x08050509 in Grunt::Wait_For_Stop ()

#4  0x0804c9b4 in Manager::Stop_Test ()

#5  0x0804e9c7 in Manager::Process_Message ()

#6  0x0804eba5 in Manager::Run ()

#7  0x0804bf81 in main ()

 

 

Thanks,

 

Eric

 

From: Vedran Degoricija [[hidden email]]
Sent: Thursday, July 11, 2013 12:30 AM
To: Chris Horneck; Bopp, Eric
Cc: [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

Hi Eric,

 

 

As Chris points out, getting a stack trace would certainly be useful. From the looks of it, your code may be waiting on the not_ready variable in Grunt::Wait_For_Stop(). But it's hard to envision the cause of it.

 

Regards,

Ved

 

From: Chris Horneck <[hidden email]>
To: "Bopp, Eric" <[hidden email]>
Cc: [hidden email]
Sent: Wednesday, July 10, 2013 2:51 PM
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

How about a gstack dump of the dynamo process when it's hung?

 

gstack <dynamo pid>

On Wed, Jul 10, 2013 at 4:11 PM, Bopp, Eric <[hidden email]> wrote:

Hi Neto,

 

I’m using the Default access specification.  I changed the transfer request size to 4 KB, but dynamo is still hanging when the test stops.

 

Thanks,

 

Eric

 

 

Additional info:  when the tests are running, the only results that get updated on the Results Display tab is CPU Utilization.  All of the other results are 0.

 

I rebuilt dynamo with  _DEBUG and _DETAILS=1.  Here is the output:

 

in while loop : Manager::Run()

Received 8 of 8 bytes from socket Taylor .

in Process_Message() : RECORD_OFF

   Getting system performance data.

   Getting system performance data.

   Getting system performance data.

CPU 0 recorded stat 0 = 2.86243

CPU 0 recorded stat 1 = 1.785

CPU 0 recorded stat 2 = 1.07743

CPU 0 recorded stat 3 = 0

CPU 0 recorded stat 4 = 0

CPU 0 recorded stat 5 = 2712.75

CPU 1 recorded stat 0 = 2.97097

CPU 1 recorded stat 1 = 1.27844

CPU 1 recorded stat 2 = 1.69253

CPU 1 recorded stat 3 = 0

CPU 1 recorded stat 4 = 0

CPU 1 recorded stat 5 = 0

TCP recorded stat 0 = 0

   Network interface 0 recorded stat 0 = 80.0273

   Network interface 0 recorded stat 1 = 0

   Network interface 0 recorded stat 2 = 0

   Network interface 1 recorded stat 0 = 657.823

   Network interface 1 recorded stat 1 = 0

   Network interface 1 recorded stat 2 = 0

   Network interface 2 recorded stat 0 = 0

   Network interface 2 recorded stat 1 = 0

   Network interface 2 recorded stat 2 = 0

   Getting system performance data.

   Getting system performance data.

   Getting system performance data.

CPU 0 recorded stat 0 = 2.86243

CPU 0 recorded stat 1 = 1.785

CPU 0 recorded stat 2 = 1.07743

CPU 0 recorded stat 3 = 0

CPU 0 recorded stat 4 = 0

CPU 0 recorded stat 5 = 2712.75

CPU 1 recorded stat 0 = 2.97097

CPU 1 recorded stat 1 = 1.27844

CPU 1 recorded stat 2 = 1.69253

CPU 1 recorded stat 3 = 0

CPU 1 recorded stat 4 = 0

CPU 1 recorded stat 5 = 0

TCP recorded stat 0 = 0

   Network interface 0 recorded stat 0 = 80.0273

   Network interface 0 recorded stat 1 = 0

   Network interface 0 recorded stat 2 = 0

   Network interface 1 recorded stat 0 = 657.819

   Network interface 1 recorded stat 1 = 0

   Network interface 1 recorded stat 2 = 0

   Network interface 2 recorded stat 0 = 0

   Network interface 2 recorded stat 1 = 0

   Network interface 2 recorded stat 2 = 0

   Stopped.

Recording stopped.

in SynchSend

Sent 8 of 8 bytes to socket Taylor .

in while loop : Manager::Run()

Received 8 of 8 bytes from socket Taylor .

in Process_Message() : STOP

Stopping...

*** HANGS HERE ***

 

From: Neto, Antonio Jose Rodrigues [mailto:[hidden email]]
Sent: Tuesday, July 09, 2013 6:49 PM
To: Bopp, Eric; [hidden email]
Subject: Re: [Iometer-user] Dynamo gets stuck at the end of testing

 

Hi Eric

 

This is neto from Brazil

 

How are you?

 

Please try to change the block size to 4096 bytes and do the test.

 

Thank you

 

All the best

 

neto

 

From: <Bopp>, Eric <[hidden email]>
Date: Tuesday, July 9, 2013 8:57 PM
To: "[hidden email]" <[hidden email]>
Subject: [Iometer-user] Dynamo gets stuck at the end of testing

 

Hello,

 

I’m looking for help with a problem I’m having running IoMeter.  The problem is that dynamo seems to hang when the first test completes.  Here is the output:

 

Command line parameter(s):

   Looking for Iometer on "10.221.220.20"

   New manager name is " taylor "

 

Sending login request...

   taylor

   10.221.220.49 (port 38153)

Successful PortTCP::Connect

  - port name: 10.221.220.20

 

*** If dynamo and iometer hangs here, please make sure

*** you use a correct -m <manager_computer_name> that

*** can ping from iometer machine. use IP if need.

   Login accepted.

Reporting drive information...

Set_Sizes: Open on "/sys/iobw.tst" failed (error No such file or directory).

Set_Sizes: Open on "/proc/sys/fs/binfmt_misc/iobw.tst" failed (error Permission denied).

  Physical drives (raw devices)...

Reporting TCP network information...

   done.

Removing worker 0.

Removing worker 0.

Worker 0 running Access Spec: 512B 0% reads 0% random

Access specifications for test defined.

Grunt: Growing grunt data buffer from 512 to 512

Worker 0 running Access Spec: 512B 0% reads 0% random

Access specifications for test defined.

Grunt: Grunt data buffer size 512 >= 512, returning

Worker 0 setting targets...

Seeding random Number Generator(1933595241135)

   Generating random data...

Starting...

   Opening targets.

Beginning to perform I/O...

   Stopped.

Stopping...

*** HANGS HERE ***

 

 

At this stage the IoMeter GUI is also frozen.

 

Here is the configuration I am using: 

 

IoMeter version 1.1.0-rc1 (iometer-1.1.0-rc1-src.tar.bz2)

IoMeter GUI is running on Windows XP. 

Dynamo is running on a linux box. Built dynamo in the linux test environment.  (make –f Makefile-linux.i386 all)

 

Thanks,

 

Eric

 

 


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user

 


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user


------------------------------------------------------------------------------
Get your SQL database under version control now!
Version control is standard for application code, but databases havent
caught up. So what steps can you take to put your SQL databases under
version control? Why should you start doing it? Read more to find out.
http://pubads.g.doubleclick.net/gampad/clk?id=49501711&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user




--
Justin W. Riggs
Windows System Administrator
(Contractor - Valdez International)

Information Technology Directorate
Interior Business Center
303-969-6623 (Office)
[hidden email]
US Department of the Interior
Office of the Secretary

Your Focus: Your Mission
Our Focus: You

------------------------------------------------------------------------------
Get your SQL database under version control now!
Version control is standard for application code, but databases havent
caught up. So what steps can you take to put your SQL databases under
version control? Why should you start doing it? Read more to find out.
http://pubads.g.doubleclick.net/gampad/clk?id=49501711&iu=/4140/ostg.clktrk
_______________________________________________
Iometer-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/iometer-user
Loading...