kernel defconfig, debugging, preemption, and very noticeable speedups

classic Classic list List threaded Threaded
43 messages Options
123
Reply | Threaded
Open this post in threaded view
|

kernel defconfig, debugging, preemption, and very noticeable speedups

Gennady Kupava
hi all, next bunch of kernel config changes.

i did series of tests with lmbench disable next part of performance-wise
options in kernel config. I archived very fast result. For example, top
average CPU usage were dropped down from 3.2% with default config kernel
to 1.6% average.

test environment:
debian on sd, andy-tracking kernel with default config
(gta02_moredrivers_defconfig) for gta02 + plus one feature changed. all
tests were run several times

Features changed as in summary report:
1. defconfig - default config
2. nodebug - debugging off : all except "Enable dynamic printk() call
support" and "timing info"
3. speed - use -O2 optimization instead of -Os
4. nopreempt - disable preemtion (CONFIG_PREEMPT)

test results:
see http://www.bsdmn.com/openmoko/summary1.txt

comments:
file operations should be ignored, variuos memcpys and cache speeds can
be used to mae sure that tests are comparable and system is ok.

iterpretation:

1. debug impact is huge - 10x in open/close test, 13% on select, 10x on
signal hadling, 2x on exec, 2x on fork, 1,8x on context switches. 50% on
pipe latency, 30% on local TCP, 30% on pipes. system feel much faster.

2. preemption disable: smaller scale speedups, but also seem very
noticeable.

3. optimising for speed have some impact in different areas, + and -, 0
in total, but much things were not tested.

comclusion:
so i propose to:
1) disable debugging for sure.
2) disable preemtion.
3) disable optimization for speed.

gennady


Reply | Threaded
Open this post in threaded view
|

Re: kernel defconfig, debugging, preemption, and very noticeable speedups

Patryk Benderz
[cut]
> so i propose to:
> 1) disable debugging for sure.
> 2) disable preemtion.
> 3) disable optimization for speed.
Hi Gennady,
really great tests. Good someone looked into it :). IMHO I agree to
disable these features, but only on stable releases. In Testing and
Unstable kernels developers should have something in log files to dig
through.

--
Patryk "LeadMan" Benderz
Linux Registered User #377521
()  ascii ribbon campaign - against html e-mail
/\  www.asciiribbon.org   - against proprietary attachments


Email secured by Check Point

Reply | Threaded
Open this post in threaded view
|

Re: kernel defconfig, debugging, preemption, and very noticeable speedups

Gennady Kupava
В Срд, 06/01/2010 в 19:40 +0100, Patryk Benderz пишет:

> [cut]
> > so i propose to:
> > 1) disable debugging for sure.
> > 2) disable preemtion.
> > 3) disable optimization for speed.
> Hi Gennady,
> really great tests. Good someone looked into it :). IMHO I agree to
> disable these features, but only on stable releases. In Testing and
> Unstable kernels developers should have something in log files to dig
> through.
>

hi Patryk,

problem with your suggestion is that I'm hever heard of stable release
for fr, now everyone is forced to use kernel several times slower that
it is in real.

i can even hardly undertand this - everyone is talking about slowness of
freerunner, but no one want to just disable debugging features to have
speedup in times.

imo, kernel dev (who are 5% of community) can compile own version to do
development - anyway they need to compile it to do development.

also, preemption cann't help developers anyhow.

gennady



Reply | Threaded
Open this post in threaded view
|

Re: kernel defconfig, debugging, preemption, and very noticeable speedups

Timo Jyrinki
In reply to this post by Gennady Kupava
2010/10/7 Gennady Kupava <[hidden email]>:
> 1. debug impact is huge - 10x in open/close test, 13% on select, 10x on
> signal hadling, 2x on exec, 2x on fork, 1,8x on context switches. 50% on
> pipe latency, 30% on local TCP, 30% on pipes. system feel much faster.
>
> 2. preemption disable: smaller scale speedups, but also seem very
> noticeable.

I can confirm that various operations are much speedier now, including
booting my Debian. This is quite awesome. What is not as awesome that
distributions etc. never questioned especially the debug features
being enabled :) But that's understandable of course given how much
stuff there is generally that only one person is actively dealing with
etc.

Could someone with access rights disable debug and pre-empt features
in gta02_moredrivers_defconfig in git.openmoko.org? I think wrong
stuff there leads to the information not spreading widely enough.

-Timo

Reply | Threaded
Open this post in threaded view
|

Re: kernel defconfig, debugging, preemption, and very noticeable speedups/ debugging

Gennady Kupava
In reply to this post by Gennady Kupava
hi, all

ok, all this proposed changes seem too much to digest in one go, let's
discuss them individually.

let's talk about debugging.

i am not true kernel hacker, but here is my thoughts, i'll be glad to be
corrected.

i see no reason why debugging features should be enabled for anyone
expect kernel developers, and kernel developers i think, should only
enable features directly related to things they are doing. for example
where is comletely no point to have such things like spinlocks, mutexes,
preemtion, irq, file system, sound, nand and other for which i cann't
guess that they are doing. do .config|grep DEBUG to see what's enabled.
many debugging features add huge overhead, and i don't understand how
they are helping anyone.

tests i did confirm that debugging overhead is huge, and i think this is
real thing which makes our device 'slow' - nothing more. so, i propose
to turn that off

gennady

В Чтв, 07/10/2010 в 01:22 +0400, Gennady Kupava пишет:

> hi all, next bunch of kernel config changes.
>
> i did series of tests with lmbench disable next part of performance-wise
> options in kernel config. I archived very fast result. For example, top
> average CPU usage were dropped down from 3.2% with default config kernel
> to 1.6% average.
>
> test environment:
> debian on sd, andy-tracking kernel with default config
> (gta02_moredrivers_defconfig) for gta02 + plus one feature changed. all
> tests were run several times
>
> Features changed as in summary report:
> 1. defconfig - default config
> 2. nodebug - debugging off : all except "Enable dynamic printk() call
> support" and "timing info"
> 3. speed - use -O2 optimization instead of -Os
> 4. nopreempt - disable preemtion (CONFIG_PREEMPT)
>
> test results:
> see http://www.bsdmn.com/openmoko/summary1.txt
>
> comments:
> file operations should be ignored, variuos memcpys and cache speeds can
> be used to mae sure that tests are comparable and system is ok.
>
> iterpretation:
>
> 1. debug impact is huge - 10x in open/close test, 13% on select, 10x on
> signal hadling, 2x on exec, 2x on fork, 1,8x on context switches. 50% on
> pipe latency, 30% on local TCP, 30% on pipes. system feel much faster.
>
> 2. preemption disable: smaller scale speedups, but also seem very
> noticeable.
>
> 3. optimising for speed have some impact in different areas, + and -, 0
> in total, but much things were not tested.
>
> comclusion:
> so i propose to:
> 1) disable debugging for sure.
> 2) disable preemtion.
> 3) disable optimization for speed.
>
> gennady
>
>



Reply | Threaded
Open this post in threaded view
|

Re: kernel defconfig, debugging, preemption, and very noticeable speedups/ debugging

Vasco Névoa
Gennady Kupava wrote:
> i see no reason why debugging features should be enabled for anyone
> expect kernel developers, and kernel developers i think, should only
> enable features directly related to things they are doing.
I second this opinion.
If the developers are afraid to lose debugging info from testers, then I
propose that they supply an optional debugging-enabled kernel package
for easy installation so that users can support debugging when the need
arises.

But in my opinion, in every testing and unstable build image things MUST
be optimized for stability AND speed.
Please remove debugging stuff from kernel ASAP.
As to compiler optimizations... well, there things get more complicated
and must be looked at carefully.

Thanks!
Vasco.

Reply | Threaded
Open this post in threaded view
|

Re: kernel defconfig, debugging, preemption, and very noticeable speedups/ debugging

Petr Vanek
>Gennady Kupava wrote:
>> i see no reason why debugging features should be enabled for anyone
>> expect kernel developers, and kernel developers i think, should only
>> enable features directly related to things they are doing.
>I second this opinion.
>If the developers are afraid to lose debugging info from testers, then
>I propose that they supply an optional debugging-enabled kernel
>package for easy installation so that users can support debugging when
>the need arises.
>
>But in my opinion, in every testing and unstable build image things
>MUST be optimized for stability AND speed. Please remove debugging
>stuff from kernel ASAP. As to compiler optimizations... well, there
>things get more complicated and must be looked at carefully.

great. if the current default kernel is -dev, why not have an
-experimental one then?

Petr


Reply | Threaded
Open this post in threaded view
|

Re: kernel defconfig, debugging, preemption, and very noticeable speedups/ debugging

Paul Fertser
In reply to this post by Vasco Névoa
Vasco Nevoa <[hidden email]> writes:
> Please remove debugging stuff from kernel ASAP.

I suspect that the most useful debugging features (that should be in
kernels all users are using) add no overhead at all.

So please show me why you think _all_ debugging features should be
disabled and not just a subset that is responsible for the slowdown.

Without that i'm not taking the responsibility to remove anything,
sorry.

--
Be free, use free (http://www.gnu.org/philosophy/free-sw.html) software!
mailto:[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: kernel defconfig, debugging, preemption, and very noticeable speedups/ debugging

Gennady Kupava
В Сбт, 09/01/2010 в 23:06 +0300, Paul Fertser пишет:

> Vasco Nevoa <[hidden email]> writes:
> > Please remove debugging stuff from kernel ASAP.
>
> I suspect that the most useful debugging features (that should be in
> kernels all users are using) add no overhead at all.
>
> So please show me why you think _all_ debugging features should be
> disabled and not just a subset that is responsible for the slowdown.
>
> Without that i'm not taking the responsibility to remove anything,
> sorry.


hi, paul

i still think that debugging options should be disabled unless we have
strong reason to enable them, so can you (or any other kernel developer)
as kernel developer name features useful for you and why users should
have them in daily usage. my initial mail were about that. please feel
free to ask me to help anyhow, here or in irc.

thanks for attention to this topic, gennady


Reply | Threaded
Open this post in threaded view
|

Re: kernel defconfig, debugging, preemption, and very noticeable speedups/ debugging

Klaus 'mrmoku' Kurzmann
On Sun, 10 Oct 2010, Gennady Kupava wrote:

> В Сбт, 09/01/2010 в 23:06 +0300, Paul Fertser пишет:
> > Vasco Nevoa <[hidden email]> writes:
> > > Please remove debugging stuff from kernel ASAP.
> >
> > I suspect that the most useful debugging features (that should be in
> > kernels all users are using) add no overhead at all.
> >
> > So please show me why you think _all_ debugging features should be
> > disabled and not just a subset that is responsible for the slowdown.
> >
> > Without that i'm not taking the responsibility to remove anything,
> > sorry.


> hi, paul

> i still think that debugging options should be disabled unless we have
> strong reason to enable them, so can you (or any other kernel developer)
> as kernel developer name features useful for you and why users should
> have them in daily usage. my initial mail were about that. please feel
> free to ask me to help anyhow, here or in irc.
I am *not* a kernel developer... though I understand why it makes a lot
of sense to have debugging enabled. Most of the time the real users will
hit a bug... and then can give usefull bug reports to the kernel guys
*if* debugging is enabled. On the other hand I understand (and agree)
that we want to get every speedup that is possible on our beloved device :)

IMO it makes a lot of sense to find out the single debug configs that
add overhead and just remove those.

> thanks for attention to this topic, gennady

--
Klaus 'mrmoku' Kurzmann

Reply | Threaded
Open this post in threaded view
|

Re: kernel defconfig, debugging, preemption, and very noticeable speedups/ debugging

Gennady Kupava
В Сбт, 09/01/2010 в 22:25 +0100, Klaus 'mrmoku' Kurzmann пишет:

> > hi, paul
>
> > i still think that debugging options should be disabled unless we have
> > strong reason to enable them, so can you (or any other kernel developer)
> > as kernel developer name features useful for you and why users should
> > have them in daily usage. my initial mail were about that. please feel
> > free to ask me to help anyhow, here or in irc.
> I am *not* a kernel developer... though I understand why it makes a lot
> of sense to have debugging enabled. Most of the time the real users will
> hit a bug... and then can give usefull bug reports to the kernel guys
> *if* debugging is enabled. On the other hand I understand (and agree)
> that we want to get every speedup that is possible on our beloved device :)
>
> IMO it makes a lot of sense to find out the single debug configs that
> add overhead and just remove those.
>

with Paul we've finally discuss the plan to change defconfig.

>I am *not* a kernel developer...

so, I failed to get attention of kernel developers to this problem :)

> Most of the time the real users will hit a bug...

we have 1 fix in branch all users use since... august i think. and this
depends on how much we pay for particular option.

> IMO it makes a lot of sense to find out the single debug configs that
> add overhead and just remove those.

I just propose to look from other side - identify options we need, as
this is debugging options, and enable them. if nobody knows why we need
some option - no reason to have it enabled.

>On the other hand I understand (and agree) that we want to get every
> speedup that is possible on our beloved device :)

mrmoku, just try kernel without debugging and you'll understand why i m
so inistent, and why people even start blaming openmoko that they didn't
do that before, forgetting for the moment that openmoko is only reason
why we have something to discuss :)

gennady


Reply | Threaded
Open this post in threaded view
|

Re: kernel defconfig, debugging, preemption, and very noticeable speedups/ debugging

Werner Almesberger
In reply to this post by Paul Fertser
Paul Fertser wrote:
> I suspect that the most useful debugging features (that should be in
> kernels all users are using) add no overhead at all.

That's also my expectation. Therefore, I'm a bit surprised by the
massive improvements reported.

Now, it could well be that there are some debug options that look
perfectly harmless (at least to me) but that have a high cost.

Another possibility would be that some options just produce a
small increase of the amount of kernel messages printed. That in
turn could, combined with the very inefficient scrolling to the
kernel's frame, buffer cause significant effects in some tests.

Another possibility is that some systems may encounter a large
number of slightly abnormal conditions that generate messages when
debugging is enabled, while other systems are quiet. Thus, not all
developers may experience the slowdown.

So, something was found, but I think a bit more work is needed to
identify what it really is. Identifying the costly debug options
will also help to be aware of the performance impact if they are
needed in the future.

I would agree that many debug options are indeed only useful when
chasing a bug, but some also produce diagnostics useful for
getting a first idea of what's happening. We would win little if
users would normally run an extremely lean kernel but get asked to
install some fat debug kernel each time they report a bug.

Related to this, it would be great if the GTA02 kernel's text
console had Glamo-accelerated scrolling. A long time ago, I
thought adding such a block move would be a nice late night
project, but I failed to elicit even the slightest response from
the Glamo. Some time later, Harald mentioned that he was giving it
a try as well, but apparently without much success either. Maybe
third time's lucky ?

By the way, Gennady, your mails may not reach some readers because
they're dated way into the future (about 9 months ahead), which is
something that normally identifies spam. And even for those who do
see the mails, it messes up their mailbox order.

Thanks,
- Werner

Reply | Threaded
Open this post in threaded view
|

Re: kernel defconfig, debugging, preemption, and very noticeable speedups/ debugging

Gennady Kupava
В Вск, 10/01/2010 в 01:52 -0300, Werner Almesberger пишет:
> Paul Fertser wrote:
> > I suspect that the most useful debugging features (that should be in
> > kernels all users are using) add no overhead at all.
>
> That's also my expectation. Therefore, I'm a bit surprised by the
> massive improvements reported.

I am really surprised that i have to spend so much time talking about
this issue, which seem very trivial for me. It's easy from my pov - if
you do spinlocks accounting - you'll get massive slowdown. if you start
debugging interrupts - you got slowdown. if you add symbols - you got 5x
memory consumption. you wish to add addtional check and debug preemption
- you'll get slowdown, if you wish to debug mutexes, you'll get
slowdown.

may be something is not so visible on desktops, but slower embedded
devices do not forgive such debugging on the side of user.

>
> Now, it could well be that there are some debug options that look
> perfectly harmless (at least to me) but that have a high cost.
>
> Another possibility would be that some options just produce a
> small increase of the amount of kernel messages printed. That in
> turn could, combined with the very inefficient scrolling to the
> kernel's frame, buffer cause significant effects in some tests.
>
> Another possibility is that some systems may encounter a large
> number of slightly abnormal conditions that generate messages when
> debugging is enabled, while other systems are quiet. Thus, not all
> developers may experience the slowdown.

no. anyone seem expierence this, see reports in community ml. have you
tried to use the kernel without debugging?

>
> So, something was found, but I think a bit more work is needed to
> identify what it really is. Identifying the costly debug options
> will also help to be aware of the performance impact if they are
> needed in the future.

it's hard to find 10% performance impact with testing individual
options, as they have stistical error, but few such 10% may provide
major slowdown and eventually run us out of cache.

>
> I would agree that many debug options are indeed only useful when
> chasing a bug, but some also produce diagnostics useful for
> getting a first idea of what's happening. We would win little if
> users would normally run an extremely lean kernel but get asked to
> install some fat debug kernel each time they report a bug.

this is normal practice in whole world - to install debugging version if
you want to help with chasing bug.

i checked debian kernel for versatile board and found that following
options are on:
CONFIG_USB_SERIAL_DEBUG=m
CONFIG_OCFS2_DEBUG_MASKLOG=y
CONFIG_DLM_DEBUG=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_KERNEL=y
CONFIG_SCHED_DEBUG=y
CONFIG_DEBUG_BUGVERBOSE=y

compare to our:
CONFIG_DEBUG_GPIO=y
CONFIG_SND_DEBUG=y
CONFIG_SND_PCM_XRUN_DEBUG=y
CONFIG_USB_S3C2410_DEBUG=y
CONFIG_REGULATOR_DEBUG=y
CONFIG_JFFS2_FS_DEBUG=0
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_SHIRQ=y
CONFIG_SCHED_DEBUG=y
CONFIG_DEBUG_PREEMPT=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_LOCK_ALLOC=y
CONFIG_DEBUG_LOCKDEP=y
CONFIG_DEBUG_SPINLOCK_SLEEP=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_SG=y
CONFIG_DYNAMIC_PRINTK_DEBUG=y
CONFIG_DEBUG_ERRORS=y
CONFIG_DEBUG_S3C_UART=2

i propose more intellegent way - identify which features we really need
and why, and turn them on. embeeded device will not forgive you adding
much accaunting also, so things which can just be 'turned on because
overhead is negligible' on desktop are better to turn off on slower
embedded device imo.

i think only options really needed are options which allow backtrace.

>
> Related to this, it would be great if the GTA02 kernel's text
> console had Glamo-accelerated scrolling. A long time ago, I
> thought adding such a block move would be a nice late night
> project, but I failed to elicit even the slightest response from
> the Glamo. Some time later, Harald mentioned that he was giving it
> a try as well, but apparently without much success either. Maybe
> third time's lucky ?

having anything accelerted is good thing, but all this debug thing is
not related to glamo or speed of kernel messages. it's hard to describe
microtests speedup like context switches or file open/close with glamo,
no way to understand why cpu usage of running top decreased 2x - no
kernel messages are produced in process. please dont think that only
boot time is changed. forget glamo in this case, i think it's ok.

> By the way, Gennady, your mails may not reach some readers because
> they're dated way into the future (about 9 months ahead), which is
> something that normally identifies spam. And even for those who do
> see the mails, it messes up their mailbox order.

urghh. fixed. hope that this is only reason :)

gennady.


Reply | Threaded
Open this post in threaded view
|

Re: kernel defconfig, debugging, preemption, and very noticeable speedups

Timo Juhani Lindfors
In reply to this post by Gennady Kupava
Gennady Kupava <[hidden email]> writes:
> 2. nodebug - debugging off : all except "Enable dynamic printk() call
> support" and "timing info"

I tested this too.

"apt-cache search nano"

went from 20 s to 14.8 s and

"emacs -f kill-emacs"

went from 3.8 s to 2.2 s. (I of course repeated these multiple times
to get caches warm.)

Reply | Threaded
Open this post in threaded view
|

Re: kernel defconfig, debugging, preemption, and very noticeable speedups/ debugging

Radek Polak
In reply to this post by Werner Almesberger
Werner Almesberger wrote:

> I would agree that many debug options are indeed only useful when
> chasing a bug, but some also produce diagnostics useful for
> getting a first idea of what's happening. We would win little if
> users would normally run an extremely lean kernel but get asked to
> install some fat debug kernel each time they report a bug.

I like the idea of two kernels - production and debugging. IMO it's the right
way to go. I'll do it for next QtMoko release.

Regards

Radek

Reply | Threaded
Open this post in threaded view
|

Re: kernel defconfig, debugging, preemption, and very noticeable speedups/ debugging

Werner Almesberger
In reply to this post by Gennady Kupava
Gennady Kupava wrote:
> I am really surprised that i have to spend so much time talking about
> this issue, which seem very trivial for me.

Thanks for having patience with those of us who are a little slow :-)

> may be something is not so visible on desktops, but slower embedded
> devices do not forgive such debugging on the side of user.

Yes, this makes perfect sense. What I'm looking for is a more detailed
quantification of the impact. What I'd expect is that many debug
options have very little impact while others make a huge difference.
We have to be careful with the latter, because they can also
invalidate experiments (see below).

I understand that things like cache thrashing may never show up unless
you change a large number of options at the same time. However, even
this is useful information, much like the mass defect in nuclear
experiments tells you something about particles you may not be able to
observe.

> no. anyone seem expierence this, see reports in community ml. have you
> tried to use the kernel without debugging?

I only changed config options when I caught them red-handed doing
something bad.

> it's hard to find 10% performance impact with testing individual
> options, as they have stistical error, but few such 10% may provide
> major slowdown and eventually run us out of cache.

Yes, death by a million gnat bites is a possibility. But have you
proven that this is the case or is it just a hypothesis ? (Like
the one that debug options have no perceptible cost, which you've
proven incorrect now.)

> this is normal practice in whole world - to install debugging version if
> you want to help with chasing bug.

What I'm afraid of is that distributions will ship radically trimmed
down kernels that don't provide any useful diagnostics. So any
suspected kernel bug would require the user to install a debug kernel.
There are a few problems with this:

- users may be reluctant to change kernels and the kernel change may
  also cause other problems (e.g., mistakes made when performing an
  unfamiliar operation) which contribute to making it harder to
  debug the problem

- today's debug kernel may not match yesterday's production kernel, so
  the symptoms need a systematic re-evaluation. This is generally very
  hard to do, particularly if it's a bug that's not trivial to
  reproduce. (Since the ones that are easy to reproduce get eliminated
  quickly, any strategy that assumes easy reproducibility will work
  great in the beginning but cause grief later.)

- if the production kernel's performance differs wildly from that of
  the debug kernel, even if using matching versions, the changes
  experience may confuse the user and result in false symptoms getting
  reported.

  Worse yet, if the problem is a race condition or an access to
  improperly initialized memory, the symptoms may simply vanish in the
  debug kernel.

That's why I'm unhappy with the recommendation to just throw out all
debugging in production kernels. I'd rather have a production kernel
that does as much debugging as possible while still delivering good
performance.

> no way to understand why cpu usage of running top decreased 2x - no
> kernel messages are produced in process.

Yes, explaining the exact underlying reasons for a quantitative change
is another can of worms :-)

> urghh. fixed. hope that this is only reason :)

Thanks a lot !

- Werner

Reply | Threaded
Open this post in threaded view
|

Re: kernel defconfig, debugging, preemption, and very noticeable speedups/ debugging

Thomas White-2
In reply to this post by Werner Almesberger
On Sun, 10 Jan 2010 01:52:22 -0300
Werner Almesberger <[hidden email]> wrote:

> Related to this, it would be great if the GTA02 kernel's text
> console had Glamo-accelerated scrolling. A long time ago, I
> thought adding such a block move would be a nice late night
> project, but I failed to elicit even the slightest response from
> the Glamo. Some time later, Harald mentioned that he was giving it
> a try as well, but apparently without much success either. Maybe
> third time's lucky ?

I thought about doing this too.  DRM gives us the ideal platform for it
because the command queue stuff is already in the kernel waiting to be
used - you just need to feed it the right commands (which can be copied
from the Xorg driver).  It certainly be nice to have (even though we
hopefully don't spend a lot of time showing the console), but I'm
suffering a lot for time at the moment. It'd be a great project for
someone who wants to learn a bit about Glamo (or graphics acceleration
in general) - no low-level scariness or NDA required, I promise...

Tom

--
Thomas White <[hidden email]>

Reply | Threaded
Open this post in threaded view
|

Re: kernel defconfig, debugging, preemption, and very noticeable speedups/ debugging

Gennady Kupava
In reply to this post by Werner Almesberger
hi all,

yesterday i had a talk about disabling debugging information on irc
with paul fertser and some shr developers. my idea were to understand
what really stands behind idea not to remove some of that options,
i went out of that discussions absolutely frustrated.

it turned out that:

1. i am wrong by definition (exactly), my opinion was completely ignored, and the whole
discussion was desclared as waste of time (exactly in this words).
2. current kernel dev (paul) is not willing to support kernel with optimizations.
3. paul also publically told that it was 'dirty trick' to publish my mail
about optimization to the 'end users', also he was very irritated by me because
i am reason for the fact that 'end users' want something from him.

my plan was to continue investigation of other options and eventually
try to fix NAND ECC handling in hardware.

now i understood that if even elementary things can't be pushed and devs
are willing to 'keep as much debugging as possible', seems that
performance is not serous issue to paul and some other devs, and i will
have no way to discuss other optimizations, i'll be just ignored or
insulted. the only constructive solution for this situation except complete give up
is to ignore this people and continue with my inverstigation.

so my current plan is:

1. don't care about andy-tracking and just post anything i've found to
this mailing list.
2. don't do anything on debug/preepmtion/optimization for size issue,
as it's definetly waste of my time.
3. i'll publish incrementally build kernel with proposed optimizations.

gennady


Reply | Threaded
Open this post in threaded view
|

Re: kernel defconfig, debugging, preemption, and very noticeable speedups/ debugging

Werner Almesberger
Gennady Kupava wrote:
> i went out of that discussions absolutely frustrated.

I saw that little flame fest a bit later. I noticed that you seemed
quite distressed, but I don't quite understand why.

Everybody agrees that you discovered something important. Everbody
also agrees that the current debugging options need changing. The
only thing where there's disagreement is about what degree of
changing your research so far justifies.

The previous assumption was that debug options are good to have and
their cost is negligible. So we used plenty of them.

You've shown that this is incorrect. However, the conclusion drawn
from this was is that all debug options are evil and they all must
be avoided.

The data you provided doesn't support such a radical conclusion.

Maybe it's clearer if I express this mathematically. D be a set of
debug options, D_{OM} be the set of of debug options the Openmoko
kernels used so far, cost(x) be the cost of debug option x where x
can be a single option used alone or a set of debug options.
Finally, \epsilon be the (non-zero) cost we would still consider
negligible.

The previous assumption was   cost(D_{OM}) < \epsilon
What you've shown is          cost(D_{OM}) \gg cost(\emptyset)
implying                      cost(D_{OM}) \gg \epsilon
Furthermore, you stated that  \sum{d in D} cost(d) \le cost{D}
The conclusion drawn is       \forall d \in D: cost(d) > \epsilon
and therefore      \nexists D \ne \emptyset: cost(D) < \epsilon

I think everybody agrees except for the last two points. And I hope
you can agree that these last points don't follow from the rest :)

It would help to clarify the issue if you could post the results of
\forall d \in D_{OM}: cost(d)
or, if you prefer to emphasize the superlinear increase of cost when
combining options, a sequence of
cost(d_0), cost({d_0, d_1}), cost({d_0, d_1, d_2}), ...

Let's replace voodoo with science, not old voodoo with new voodoo :-)

Thanks,
- Werner

Reply | Threaded
Open this post in threaded view
|

Re: kernel defconfig, debugging, preemption, and very noticeable speedups/ debugging

Paul Fertser
In reply to this post by Gennady Kupava
Hi,

To avoid any further confusion i'd like to clarify some points, please
bear with me.

Also, please keep in mind neither me, nor Gennady or other
participants of the discussion are native speakers so it is quite
possible something very important was "lost in translation". I really
see no reason whatsoever for any kind of emotional response or
arguments.

Gennady Kupava <[hidden email]> writes:
> 1. i am wrong by definition (exactly), my opinion was completely
> ignored, and the whole discussion was desclared as waste of time
> (exactly in this words).

It wasn't me who claimed that and i disagree with this idea. I'm not
sure but probably you misunderstood some statement or you was
misunderstood.

> 2. current kernel dev (paul) is not willing to support kernel with
> optimizations.

I'm not "current kernel dev", that's first. I'm still committed to
making OM kernel better if and when i can do that. Optimizations are
very essential and important. I'm ready to participate in optimization
work.

> 3. paul also publically told that it was 'dirty trick' to publish my mail
> about optimization to the 'end users', also he was very irritated by me because
> i am reason for the fact that 'end users' want something from him.

I wasn't irritated by you at all. I was irritated by the end-users. I
never claimed that you had any malice intentions or purposely used
"dirtry tricks". It was my impression that the result of the letter by
some user (and it wasn't you) was similar to that of a "dirty trick",
i.e. end-users demanding to do something kernel-level ASAP is unfun.

Sorry for not explaining it clearly the time we talked :(

> my plan was to continue investigation of other options and eventually
> try to fix NAND ECC handling in hardware.

That's a good plan imho.

> now i understood that if even elementary things can't be pushed and devs
> are willing to 'keep as much debugging as possible',

As much as possible unless it hurts performance. And no, things can
and are being pushed.

> seems that performance is not serous issue to paul and some other
> devs

I'm afraid this is a wrong impression.

> and i will have no way to discuss other optimizations, i'll be just
> ignored or insulted.

I hope it will never be the case again.

I think everybody's interested in seeing as much results wrt
optimization as possible, it's great you're still interested in it and
i hope the community will find a way to collaborate with you fruitfully.

--
Be free, use free (http://www.gnu.org/philosophy/free-sw.html) software!
mailto:[hidden email]

123