Optimization help - reading out of /proc on Solaris

[Daniel Berger]

Hi all,

Ruby 1.8.6
Solaris 10

I recently converted a C extension to get process table information on
Solaris into a pure Ruby. I knew it would be slower, I just didn't
realize how _much_ slower it would be. I was expecting the pure Ruby
version to be about 1/10th as fast. Instead, it's about 1/70th as
fast. Anticipating the, "Is it fast enough?" question, my answer is,
"I'm not sure". Besides, tuning can be fun.

Anyway, below is the code. I ran it through the profiler, but the top
two most costly ops were Dir.foreach, which I don't see any way to
optimize*, and the loop that gathers environment information, which I
again see no way to optimize.

# sunos.rb
#
# A pure Ruby version of sys-proctable for SunOS 5.8 or later
#--
# Directories under /proc on Solaris 2.8+

# The Sys module serves as a namespace only.
module Sys

# The ProcTable class encapsulates process table information.
class ProcTable

# The version of the sys-proctable library
VERSION = '0.8.0'

private

PRNODEV = -1 # non-existent device

FIELDS = [
:flag, # process flags (deprecated)
:nlwp, # number of active lwp's in the process
id, # unique process id
pid, # process id of parent
gid, # pid of session leader
:sid, # session id
:uid, # real user id
:euid, # effective user id
:gid, # real group id
:egid, # effective group id
:addr, # address of the process
:size, # size of process in kbytes
:rssize, # resident set size in kbytes
:ttydev, # tty device (or PRNODEV)
ctcpu, # % of recent cpu used by all lwp's
ctmem, # % of system memory used by process
:start, # absolute process start time
:time, # usr + sys cpu time for this process
:ctime, # usr + sys cpu time for reaped children
:fname, # name of the exec'd file
sargs, # initial characters argument list
:wstat, # if a zombie, the wait status
:argc, # initial argument count
:argv, # address of initial argument vector
:envp, # address of initial environment vector
:dmodel, # data model of the process
:taskid, # task id
rojid, # project id
:nzomb, # number of zombie lwp's in the process
oolid, # pool id
:zoneid, # zone id
:contract, # process contract
:lwpid, # lwp id
:wchan, # wait address for sleeping lwp
:stype, # synchronization event type
:state, # numeric lwp state
:sname, # printable character for state
:nice, # nice for cpu usage
:syscall, # system call number (if in syscall)
ri, # priority
:clname, # scheduling class name
:name, # name of system lwp
npro, # processor which last ran thsi lwp
:bindpro, # processor to which lwp is bound
:bindpset, # processor set to which lwp is bound
:count, # number of contributing lwp's
:tstamp, # current time stamp
:create, # process/lwp creation time stamp
:term, # process/lwp termination time stamp
:rtime, # total lwp real (elapsed) time
:utime, # user level cpu time
:stime, # system call cpu time
:ttime, # other system trap cpu time
:tftime, # text page fault sleep time
:dftime, # text page fault sleep time
:kftime, # kernel page fault sleep time
:ltime, # user lock wait sleep time
:slptime, # all other sleep time
:wtime, # wait-cpu (latency) time
:stoptime, # stopped time
:minf, # minor page faults
:majf, # major page faults
:nswap, # swaps
:inblk, # input blocks
ublk, # output blocks
:msnd, # messages sent
:mrcv, # messages received
:sigs, # signals received
:vctx, # voluntary context switches
:ictx, # involuntary context switches
:sysc, # system calls
:ioch, # chars read and written
ath, # array of symbolic link paths from /proc/<pid>/
pid
:contracts, # array symbolic link paths from /proc/<pid>/
contracts
:fd, # array of used file descriptors
:cmd_args, # array of command line arguments
:environ # hash of environment associated with the process
]

public

ProcTableStruct = Struct.new("ProcTableStruct", *FIELDS)

# In block form, yields a ProcTableStruct for each process entry
that you
# have rights to. This method returns an array of
ProcTableStruct's in
# non-block form.
#
# If a +pid+ is provided, then only a single ProcTableStruct is
yielded or
# returned, or nil if no process information is found for that
+pid+.
#
# Example:
#
# # Iterate over all processes
# ProcTable.ps do |proc_info|
# p proc_info
# end
#
# # Print process table information for only pid 1001
# p ProcTable.ps(1001)
#
def self.ps(pid = nil)
array = block_given? ? nil : []

Dir.foreach("/proc") do |file|
next if file =~ /\D/ # Skip non-numeric entries under /
proc

# Only return information for a given pid, if provided
if pid
next unless file.to_i == pid
end

# Skip over any entries we don't have permissions to read
begin
psinfo = IO.read("/proc/#{file}/psinfo")
rescue StandardError, Errno::EACCES
next
end

struct = ProcTableStruct.new

struct.flag = psinfo[0,4].unpack("i")[0] # pr_flag
struct.nlwp = psinfo[4,4].unpack("i")[0] # pr_nlwp
struct.pid = psinfo[8,4].unpack("i")[0] # pr_pid
struct.ppid = psinfo[12,4].unpack("i")[0] # pr_ppid
struct.pgid = psinfo[16,4].unpack("i")[0] # pr_pgid
struct.sid = psinfo[20,4].unpack("i")[0] # pr_sid
struct.uid = psinfo[24,4].unpack("i")[0] # pr_uid
struct.euid = psinfo[28,4].unpack("i")[0] # pr_euid
struct.gid = psinfo[32,4].unpack("i")[0] # pr_gid
struct.egid = psinfo[36,4].unpack("i")[0] # pr_egid
struct.addr = psinfo[40,4].unpack("L")[0] # pr_addr

struct.size = psinfo[44,4].unpack("L")[0] * 1024 #
pr_size
struct.rssize = psinfo[48,4].unpack("L")[0] * 1024 #
pr_rssize

# skip pr_pad1

# TODO: Convert this to a human readable string somehow
struct.ttydev = psinfo[56,4].unpack("i")[0] # pr_ttydev

# pr_pctcpu
struct.pctcpu = (psinfo[60,2].unpack("S")[0] * 100).to_f /
0x8000

# pr_pctmem
struct.pctmem = (psinfo[62,2].unpack("S")[0] * 100).to_f /
0x8000

struct.start = Time.at(psinfo[64,8].unpack("L")[0]) #
pr_start
struct.time = psinfo[72,8].unpack("L")[0] #
pr_time
struct.ctime = psinfo[80,8].unpack("L")[0] #
pr_ctime

struct.fname = psinfo[88,16].strip # pr_fname
struct.psargs = psinfo[104,80].strip # pr_psargs
struct.wstat = psinfo[184,4].unpack("i")[0] # pr_wstat
struct.argc = psinfo[188,4].unpack("i")[0] # pr_argc
struct.argv = psinfo[192,4].unpack("L")[0] # pr_argv
struct.envp = psinfo[196,4].unpack("L")[0] # pr_envp
struct.dmodel = psinfo[200,1].unpack("C")[0] # pr_dmodel

# skip pr_pad2

struct.taskid = psinfo[204,4].unpack("i")[0] # pr_taskid
struct.projid = psinfo[208,4].unpack("i")[0] #
pr_projectid
struct.nzomb = psinfo[212,4].unpack("i")[0] # pr_nzomb
struct.poolid = psinfo[216,4].unpack("i")[0] # pr_poolid
struct.zoneid = psinfo[220,4].unpack("i")[0] # pr_zoneid
struct.contract = psinfo[224,4].unpack("i")[0] #
pr_contract

# skip pr_filler

### lwpsinfo struct info

# skip pr_flag

struct.lwpid = psinfo[236,4].unpack("i")[0] # pr_lwpid

# skip pr_addr

struct.wchan = psinfo[244,4].unpack("L")[0] # pr_wchan
struct.stype = psinfo[248,1].unpack("C")[0] # pr_stype
struct.state = psinfo[249,1].unpack("C")[0] # pr_state
struct.sname = psinfo[250,1] # pr_sname
struct.nice = psinfo[251,1].unpack("C")[0] # pr_nice
struct.syscall = psinfo[252,2].unpack("S")[0] # pr_syscall

# skip pr_oldpri
# skip pr_cpu

struct.pri = psinfo[256,4].unpack("i")[0] # pr_syscall

# skip pr_pctcpu
# skip pr_pad
# skip pr_start
# skip pr_time

struct.clname = psinfo[280,8].strip # pr_clname
struct.name = psinfo[288,16].strip # pr_name
struct.onpro = psinfo[304,4].unpack("i")[0] # pr_onpro
struct.bindpro = psinfo[308,4].unpack("i")[0] #
pr_bindpro
struct.bindpset = psinfo[308,4].unpack("i")[0] #
pr_bindpset

# Get the full command line out of /proc/<pid>/as.
begin
fd = File.open("/proc/#{file}/as")

fd.sysseek(struct.argv, IO::SEEK_SET)
address = fd.sysread(struct.argc * 4).unpack("L")[0]

struct.cmd_args = []

0.upto(struct.argc - 1){ |i|
fd.sysseek(address, IO::SEEK_SET)
data = fd.sysread(128)[/^[^\0]*/] # Null strip
struct.cmd_args << data
address += data.length + 1 # Add 1 for the space
}

# Get the environment hash associated with the process.
struct.environ = {}

fd.sysseek(struct.envp, IO::SEEK_SET)

env_address = fd.sysread(128).unpack("L")[0]

loop do
fd.sysseek(env_address, IO::SEEK_SET)
data = fd.sysread(1024)[/^[^\0]*/] # Null strip
break if data.empty?
key, value = data.split('=')
struct.environ[key] = value
env_address += data.length + 1 # Add 1 for the space
end
rescue Errno::EACCES, Errno::EOVERFLOW, EOFError
# Skip this if we don't have proper permissions, if
there's
# no associated environment, or if there's a largefile
issue.
ensure
fd.close if fd
end

### struct prusage

begin
prusage = 0.chr * 512
prusage = IO.read("/proc/#{file}/usage")

# skip pr_lwpid
struct.count = prusage[4,4].unpack("i")[0] #
pr_count
struct.tstamp = prusage[8,8].unpack("L")[0] #
pr_tstamp
struct.create = prusage[16,8].unpack("L")[0] #
pr_create
struct.term = prusage[24,8].unpack("L")[0] #
pr_term
struct.rtime = prusage[32,8].unpack("L")[0] #
pr_rtime
struct.utime = prusage[40,8].unpack("L")[0] #
pr_utime
struct.stime = prusage[48,8].unpack("L")[0] #
pr_stime
struct.ttime = prusage[56,8].unpack("L")[0] #
pr_ttime
struct.tftime = prusage[64,8].unpack("L")[0] #
pr_tftime
struct.dftime = prusage[72,8].unpack("L")[0] #
pr_dftime
struct.kftime = prusage[80,8].unpack("L")[0] #
pr_kftime
struct.ltime = prusage[88,8].unpack("L")[0] #
pr_ltime
struct.slptime = prusage[96,8].unpack("L")[0] #
pr_slptime
struct.wtime = prusage[104,8].unpack("L")[0] #
pr_wtime
struct.stoptime = prusage[112,8].unpack("L")[0] #
pr_stoptime
struct.minf = prusage[120,4].unpack("L")[0] #
pr_minf
struct.majf = prusage[124,4].unpack("L")[0] #
pr_majf
struct.nswap = prusage[128,4].unpack("L")[0] #
pr_nswap
struct.inblk = prusage[128,4].unpack("L")[0] #
pr_inblk
struct.oublk = prusage[128,4].unpack("L")[0] #
pr_oublk
struct.msnd = prusage[128,4].unpack("L")[0] #
pr_msnd
struct.mrcv = prusage[128,4].unpack("L")[0] #
pr_mrcv
struct.sigs = prusage[128,4].unpack("L")[0] #
pr_sigs
struct.vctx = prusage[128,4].unpack("L")[0] #
pr_vctx
struct.ictx = prusage[128,4].unpack("L")[0] #
pr_ictx
struct.sysc = prusage[128,4].unpack("L")[0] #
pr_sysc
struct.ioch = prusage[128,4].unpack("L")[0] #
pr_ioch
rescue Errno::EACCES
# Do nothing if we lack permissions. Just move on.
end

# Information from /proc/<pid>/path. This is represented
as a hash,
# with the symbolic link name as the key, and the file it
links to
# as the value, or nil if it cannot be found.
#--
# Note that cwd information can be gathered from here,
too.
struct.path = {}

Dir["/proc/#{file}/path/*"].each{ |entry|
link = File.readlink(entry) rescue nil
struct.path[File.basename(entry)] = link
}

# Information from /proc/<pid>/contracts. This is
represented as
# a hash, with the symbolic link name as the key, and the
file
# it links to as the value.
struct.contracts = {}

Dir["/proc/#{file}/contracts/*"].each{ |entry|
link = File.readlink(entry) rescue nil
struct.contracts[File.basename(entry)] = link
}

# Information from /proc/<pid>/fd. This returns an array
of
# numeric file descriptors used by the process.
struct.fd = Dir["/proc/#{file}/fd/*"].map{ |f|
File.basename(f).to_i }

if block_given?
yield struct
else
array << struct
end
end

pid ? array[0] : array
end
end
end

Thanks,

Dan

* I tried tossing threads at it in a one-thread-per-directory
approach, but they didn't get along with IO.read in MRI, and seemed to
provide no real speed benefit with JRuby.

 

[Brian Candler]

Anyway, below is the code. I ran it through the profiler, but the top

two most costly ops were Dir.foreach, which I don't see any way to
optimize*, and the loop that gathers environment information, which I
again see no way to optimize.

Could you post your profiling? If you run using "time", how much user
CPU versus system CPU are you using?

Have you tried using Dir.open.each instead of Dir["/foo/*"]? Maybe
globbing is expensive.

Your environment loop does a fresh sysread(1024) for each var=val pair,
even if you've only consumed (say) 7 bytes from the previous call. You
would make many fewer system calls if you read a big chunk and chopped
it up afterwards. This may also avoid non-byte-aligned reads.

I would also be tempted to write one long unpack instead of lots of
string slicing and unpacking. The overhead here may be negligible, but
the code may end up being smaller and simpler. e.g.

struct = ProcTableStruct.new(*psinfo.unpack(<<PATTERN))
i i i i
i i i i
i i L L
L x4i ss
...etc
PATTERN

Perhaps you could combine it with your struct building, e.g.

FIELDS = [
[:flag,"i"], # process flags (deprecated)
[:nlwp,"i"], # number of active lwp's in the process
...
[:size,"s"], # size of process in kbytes
[:rssize,"s"], # resident set size in kbytes
[nil,"X4"], # skip pr_pad1
... etc

HTH,

Brian.

 

[Robert Klemme]

I recently converted a C extension to get process table information on
Solaris into a pure Ruby. I knew it would be slower, I just didn't
realize how _much_ slower it would be. I was expecting the pure Ruby
version to be about 1/10th as fast. Instead, it's about 1/70th as
fast. Anticipating the, "Is it fast enough?" question, my answer is,
"I'm not sure". Besides, tuning can be fun.

Anyway, below is the code. I ran it through the profiler, but the top
two most costly ops were Dir.foreach, which I don't see any way to
optimize*, and the loop that gathers environment information, which I
again see no way to optimize.

You can optimize the loop body. Profiler output takes a while to get used to.

Few things that caught my attention:

You can combine the first two "next" in one. Also matching with the
RX on the left side is faster AFAIK.

Try to use as few psinfo.unpack as possible, i.e. ideally only 1.

Use the block form of File.open.

Do not use sysread/syswrite/sysseek unless you have to (most of the
time you don't).

You can replace all but the first checks for block_given? with
"array". Might be faster.

Have fun!

robert

 

[Ken Bloom]

Anyway, below is the code. I ran it through the profiler, but the top
two most costly ops were Dir.foreach, which I don't see any way to
optimize*, and the loop that gathers environment information, which I
again see no way to optimize.

Could you post your profiling? If you run using "time", how much user
CPU versus system CPU are you using?

Have you tried using Dir.open.each instead of Dir["/foo/*"]? Maybe
globbing is expensive.

Your environment loop does a fresh sysread(1024) for each var=val pair,
even if you've only consumed (say) 7 bytes from the previous call. You
would make many fewer system calls if you read a big chunk and chopped
it up afterwards. This may also avoid non-byte-aligned reads.

I would also be tempted to write one long unpack instead of lots of
string slicing and unpacking. The overhead here may be negligible, but
the code may end up being smaller and simpler. e.g.

struct = ProcTableStruct.new(*psinfo.unpack(<<PATTERN))
i i i i
i i i i
i i L L
L x4i ss
..etc
PATTERN

Perhaps you could combine it with your struct building, e.g.

FIELDS = [
[:flag,"i"], # process flags (deprecated)
[:nlwp,"i"], # number of active lwp's in the process
...
[:size,"s"], # size of process in kbytes
[:rssize,"s"], # resident set size in kbytes
[nil,"X4"], # skip pr_pad1
... etc

HTH,

Brian.

Here's a concept for metaprogramming that that I was able to generate
mostly by running regexes to transform the code. I've only tackled
/proc/#{file}/psinfo, but it should be fairly simple to extend to
the other files as well

# The Sys module serves as a namespace only.
module Sys

# The ProcTable class encapsulates process table information.
class ProcTable

# The version of the sys-proctable library
VERSION = '0.8.0'

private

PRNODEV = -1 # non-existent device


#Dissecting the format of this, we have a symbol mapping to an unpack format string segment
#the @ sign followed by a number indicates the offset in the string, and the text following that number is the format
#of the data to unpack
FIELDS=[
[:flag , "@0 i"],
[:nlwp , "@4 i"],
[id , "@8 i"],
[pid , "@12 i"],
[gid , "@16 i"],
[:sid , "@20 i"],
[:uid , "@24 i"],
[:euid , "@28 i"],
[:gid , "@32 i"],
[:egid , "@36 i"],
[:addr , "@40 L"],
[:size , "@44 L"],
[:rssize , "@48 L"],
[:ttydev , "@56 i"],
[ctcpu , "@60 S"],
[ctmem , "@62 S"],
[:start , "@64 L"],
[:time , "@72 L"],
[:ctime , "@80 L"],
#note that the A format specifier automatically does what the #strip method does
#so I don't have to call .strip in the ps method
[:fname , "@88 A16"],
[sargs , "@104 A80"],
[:wstat , "@184 i"],
[:argc , "@188 i"],
[:argv , "@192 L"],
[:envp , "@196 L"],
[:dmodel , "@200 C"],
[:taskid , "@204 i"],
[rojid , "@208 i"],
[:nzomb , "@212 i"],
[oolid , "@216 i"],
[:zoneid , "@220 i"],
[:contract , "@224 i"],
[:lwpid , "@236 i"],
[:wchan , "@244 L"],
[:stype , "@248 C"],
[:state , "@249 C"],
[:sname , "@250 a1"],
[:nice , "@251 C"],
[:syscall , "@252 S"],
[ri , "@256 i"],
[:clname , "@280 A8"],
[:name , "@288 A16"],
[npro , "@304 i"],
[:bindpro , "@308 i"],
[:bindpset , "@308 i"]
]

field_names,format_strings=FIELDS.transpose

eval <<-"end;"
def first_pass_fill string
struct=ProcTableStruct.new

#{ field_names.collect{|x| "struct.#{x}"}.join(", ") } = string.unpack "#{format_strings.join ' '}"
end
end;

#repeat the above with a new array instead of FIELDS and a new method name
#for any other file you want to unpack this way

=begin
This eval will define a function with the following code. The arrays and
metaprogramming are just an easier way to manage the format string and
fieldnames that you can understand them when maintenence time comes around.

def first_pass_fill string
struct=ProcTableStruct.new

struct.flag, struct.nlwp, struct.pid, struct.ppid, struct.pgid,
struct.sid, struct.uid, struct.euid, struct.gid, struct.egid, struct.addr,
struct.size, struct.rssize, struct.ttydev, struct.pctcpu, struct.pctmem,
struct.start, struct.time, struct.ctime, struct.fname, struct.psargs,
struct.wstat, struct.argc, struct.argv, struct.envp, struct.dmodel,
struct.taskid, struct.projid, struct.nzomb, struct.poolid, struct.zoneid,
struct.contract, struct.lwpid, struct.wchan, struct.stype, struct.state,
struct.sname, struct.nice, struct.syscall, struct.pri, struct.clname,
struct.name, struct.onpro, struct.bindpro, struct.bindpset = string.unpack "@0
i @4 i @8 i @12 i @16 i @20 i @24 i @28 i @32 i @36 i @40 L @44 L @48 L @56 i
@60 S @62 S @64 L @72 L @80 L @88 A16 @104 A80 @184 i @188 i @192 L @196 L @200
C @204 i @208 i @212 i @216 i @220 i @224 i @236 i @244 L @248 C @249 C @250 a1
@251 C @252 S @256 i @280 A8 @288 A16 @304 i @308 i @308 i"
end
=end

public

ProcTableStruct = Struct.new("ProcTableStruct", *field_names)

#if you have multiple files you're reading from with their field names
#in multiple different variables, you'll want to replace field_names
#with some array concatentation


# In block form, yields a ProcTableStruct for each process entry that you
# have rights to. This method returns an array of ProcTableStruct's in
# non-block form.
#
# If a +pid+ is provided, then only a single ProcTableStruct is yielded or
# returned, or nil if no process information is found for that +pid+.
#
# Example:
#
# # Iterate over all processes
# ProcTable.ps do |proc_info|
# p proc_info
# end
#
# # Print process table information for only pid 1001
# p ProcTable.ps(1001)
#
def self.ps(pid = nil)
array = block_given? ? nil : []
Dir.foreach("/proc") do |file|
next if file =~ \D # Skip non-numeric entries under / proc

# Only return information for a given pid, if provided
if pid
next unless file.to_i == pid
end

# Skip over any entries we don't have permissions to read
begin
psinfo = IO.read("/proc/#{file}/psinfo")
rescue StandardError, Errno::EACCES
next
end


#the first pass fill just gets the raw data and unpacks it
struct = first_pass_fill psinfo
#now we do the transformations we need on the few fields that need it
struct.pctcpu= (struct.pctcpu*100).to_f / 0x8000
struct.pctmem= (struct.pctmem*100).to_f / 0x8000
struct.start=Time.at(struct.start)
#the fields that needed stripping were handled by unpack

#repeat the above for other files that we need to deal with

if block_given?
yield struct
else
array << struct
end
end

pid ? array[0] : array
end
end
end

 

[Ken Bloom]

Anyway, below is the code. I ran it through the profiler, but the top
two most costly ops were Dir.foreach, which I don't see any way to
optimize*, and the loop that gathers environment information, which I
again see no way to optimize.

Could you post your profiling? If you run using "time", how much user
CPU versus system CPU are you using?

Have you tried using Dir.open.each instead of Dir["/foo/*"]? Maybe
globbing is expensive.

Your environment loop does a fresh sysread(1024) for each var=val pair,
even if you've only consumed (say) 7 bytes from the previous call. You
would make many fewer system calls if you read a big chunk and chopped
it up afterwards. This may also avoid non-byte-aligned reads.

I would also be tempted to write one long unpack instead of lots of
string slicing and unpacking. The overhead here may be negligible, but
the code may end up being smaller and simpler. e.g.

Method calls are expensive. Right now he's got 3 per field (array
indexing, unpack, and assignment). He could get that down to 1 per
field (assignment only) pretty easily by following your suggestion.