วันพุธ, พฤษภาคม 24, 2549

find และ xargs

เค้าว่าใช้ find ... | (ไปป์ไปที่) xargs ... นั้นมันให้เพอร์ฟอร์มานซ์ดีกว่าใช้ find ... -exec ...
เพราะว่า find -exec มัน fork เยอะ
(อ้างอิง)
http://www.sunmanagers.org/pipermail/summaries/2005-March/006255.html

Thanks very much for the quick responses - herewith a summary of what I learned:

The problem is not one of recursion limit, but one of (in)efficiency.

It is much more efficient to use find | xargs than find -exec {} \;

"find ... -exec command {} \;" runs the command once for each name.

If find identifies 10000 files, using -exec grep ... as an example,
find would fork off a child copy of itself, the copy would become a
grep working on a single file. Meanwhile the original parent find
would sleep waiting for its child (grep) to finish before searching
for the next file that meets the find criteria. That means 10000
executions of grep and pauses of find.

"xargs command" reads names on its standard input, and feeds them in
bunches to command, so that command is run fewer times (only once, in
most cases). So, if we use find | xargs grep ..., the find is able to
work "non-stop" filling the pipe with found file names. xargs
collects groups of about 20 - 50 names and does a single grep for the
collection. Find doesn't pause and you only do about 200 greps, not
10000. There isn't a forked process for every single file.

The weaknesses of xargs are (1) it can be confused by "funny"
filenames (which is why Gnu has find -print0 |xargs -0, or you can
pipe through sed to add backslashes everywhere), and (2) it can feed
zero arguments to command which might then
just sit there waiting (this is why Gnu xargs has -r which means:
don't run command if stdin is empty).

These days, you can use "find ... -exec command {} +" which groups
arguments together. However, I found that this did not give
satisfactory grep results when invoked as:

find /usr/include -type f -exec grep NFS_VERSION {} \+ -print (or
without the escaped +)

The following examples show the time difference (and slightly more
useful output of xargs):

$ time find /usr/include -type f -exec grep NFS_VERSION {} \; -print
#define NFS_VERSION ((rpcvers_t)2)
/usr/include/nfs/nfs.h
#define NFS_VERSION 2
/usr/include/rpcsvc/nfs_prot.h
version NFS_VERSION {
/usr/include/rpcsvc/nfs_prot.x

real 0m9.883s
user 0m3.340s
sys 0m5.900s
$ time find /usr/include -type f|xargs grep NFS_VERSION
/usr/include/nfs/nfs.h:#define NFS_VERSION ((rpcvers_t)2)
/usr/include/rpcsvc/nfs_prot.h:#define NFS_VERSION 2
/usr/include/rpcsvc/nfs_prot.x: version NFS_VERSION {

real 0m0.677s
user 0m0.380s
sys 0m0.310s

Thanks,

Steve Nelson


ตัวอย่างการใช้ เช่น เราจะหา md5 checksum ของไฟล์หลายๆไฟล์แบบเวียนเกิด recursively ก็ใช้คำสั่ง

$ find . -type f \! -name "._*" -print0 | xargs -0 md5sum

เอาเฉพาะ regular file
อันที่ไม่ได้ขึ้นต้นด้วย ._ ใน Mac OS มันชอบมีไฟล์ที่ขึ้นต้นด้วย ._ เยอะ
print ออกมาโดยใส่ null (\0) แทน whitespace ธรรมดา เพราะ xargs มันไม่ค่อยฉลาดตรงนี้

t I jojo$ time find . \! -name ".*" -print0 | xargs -0 md5sum
17c6d0eb1e725ef16e78bfd8c99219c7 ./01 ใคร.m4a
a0bbe221525288210e0a1cdb4f03eb95 ./02 ผมแอบชอบคุณอยู่.m4a
df68e7fc0984fa7dbe1172d476300c2b ./03 คนข้างล่าง.m4a
ccf9def29b9027d62f597c6e31e07066 ./04 Live & Learn.m4a
a1be585bd5e1b304aec8905a1959621c ./05 นานแค่ไหน.m4a
9759b2c841786f07d70260975932a4d7 ./06 ล้มบ้างก็ได้.m4a
214db53c2c2f519f59cc3b24fd6eb360 ./07 เหมือนเคย.m4a
3945193b6ecb2497cf04acb5f26d5ea3 ./08 พอ.m4a
7d2d16c919de20d1e3d174699f0aa821 ./09 หัวใจผูกกัน.m4a
babba2eb62de6f080280ec896016fe5c ./10 Who,What,When,Where,Why.m4a
55f6de761c829ea1adc227f14ad181f6 ./11 ต่างมุม.m4a
afe989790d19fd36b8f8c28fa8e555c4 ./12 เพลงรัก.m4a

real 0m56.337s
user 0m12.174s
sys 0m1.962s

ไม่มีความคิดเห็น: