Linux: mmap()-ing instead of read()-ing input files
Posted: 2006-07-14T12:05:03-07:00
Hi!
I faced a situation with convert when it consumes 900+ MB quickly, when only 58 MB of input PNGs were present. I doubt PNGs have such compression rate.
In fact, a lot of mem could be saved by mmap() of input files instead of open()/malloc()/read()
From my experience, it is twice as fast for larger files. Kernel will pagein files and then pageout those pages which are probably uncompressed and we no longer need them. besides, ti is space efficient, as no memory is allocated by malloc(), kernel does this job for us through virtual memory subsystem.
Is it too high of a complexity to implement in IM 6.2.6+?
I have experimented with this Unix call, and of course there can and should always be fallback procedure for "classic" read() in case mmap() fails. I use it this way in my clustering tools, and it works.
But of course, all system libraries are mapped that way.
Thanks for considering this!
Best regards,
Marvin
I faced a situation with convert when it consumes 900+ MB quickly, when only 58 MB of input PNGs were present. I doubt PNGs have such compression rate.
In fact, a lot of mem could be saved by mmap() of input files instead of open()/malloc()/read()
From my experience, it is twice as fast for larger files. Kernel will pagein files and then pageout those pages which are probably uncompressed and we no longer need them. besides, ti is space efficient, as no memory is allocated by malloc(), kernel does this job for us through virtual memory subsystem.
Is it too high of a complexity to implement in IM 6.2.6+?
I have experimented with this Unix call, and of course there can and should always be fallback procedure for "classic" read() in case mmap() fails. I use it this way in my clustering tools, and it works.
But of course, all system libraries are mapped that way.
Thanks for considering this!
Best regards,
Marvin