bugs.gw.com
Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0000164filegeneralpublic2011-12-31 22:462013-03-25 14:07
ReporterMike Frysinger 
Assigned ToChristos Zoulas 
PrioritynormalSeverityminorReproducibilityalways
StatusresolvedResolutionfixed 
PlatformOSOS Version
Product Version 
Target VersionFixed in Version5.18 
Summary0000164: awk BEGIN rule slows down parsing of ascii files significantly
Descriptionthe magic/Magdir/commands file got this rule somewhat recently:
  !:mime text/x-awk
  0 regex =^\\s*BEGIN\\s*[{] awk script text

unfortunately, this slows down things significantly when working with large ASCII files (over 5KiB for example)
Additional Informationcreate a file with 10KB newlines:
  $ dd ibs=10000 count=1 if=/dev/zero | tr '\0' '\n' > newlines

run file w/out the BEGIN regex (in multi- or single- byte locale):
  $ time file newlines
  file: ASCII text
  real 0m0.053s
  user 0m0.028s
  sys 0m0.008s

run file w/the BEGIN regex (in a multibyte locale):
  $ LC_ALL=en_US.UTF8; time LC_ALLfile newlines
  file: ASCII text
  real 0m1.580s
  user 0m1.556s
  sys 0m0.008s
that's ~3000% increase in time :(

run file w/the BEGIN regex (in a single byte locale):
  $ LC_ALL=C; time LC_ALLfile newlines
  file: ASCII text
  real 0m0.239s
  user 0m0.208s
  sys 0m0.012s
that's 0000250:0000450% increase in time :(
TagsNo tags attached.
Attached Filespatch file icon 0001-fix-possible-DoS.patch [^] (1,024 bytes) 2013-03-25 13:18 [Show Content]

- Relationships

-  Notes
(0000283)
Christos Zoulas (manager)
2012-06-21 01:19

Perhaps change regex to regex/1024 is good enough.
(0000434)
Carsten Wolff (reporter)
2013-03-25 13:18
edited on: 2013-03-25 13:21

The attached patch fixes the issue for us. Although, as mentioned in the patch discription, it is a compromise between effectivity and efficiency.

(0000435)
Christos Zoulas (manager)
2013-03-25 14:07

Applied.

- Issue History
Date Modified Username Field Change
2011-12-31 22:46 Mike Frysinger New Issue
2012-06-21 01:19 Christos Zoulas Note Added: 0000283
2012-06-21 01:19 Christos Zoulas Assigned To => Christos Zoulas
2012-06-21 01:19 Christos Zoulas Status new => feedback
2013-03-25 13:18 Carsten Wolff Note Added: 0000434
2013-03-25 13:18 Carsten Wolff File Added: 0001-fix-possible-DoS.patch
2013-03-25 13:21 Carsten Wolff Note Edited: 0000434
2013-03-25 14:06 Christos Zoulas Assigned To Christos Zoulas =>
2013-03-25 14:07 Christos Zoulas Status feedback => resolved
2013-03-25 14:07 Christos Zoulas Fixed in Version => 5.18
2013-03-25 14:07 Christos Zoulas Resolution open => fixed
2013-03-25 14:07 Christos Zoulas Assigned To => Christos Zoulas
2013-03-25 14:07 Christos Zoulas Note Added: 0000435


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker