| --- | Log | opened Wed Mar 08 00:00:34 2006 |
| 00:36 | |-| | Jeremy [~ph33r@crstn.bc.ipv6.smart-serv.net] has quit [Ping timeout: 480 seconds] |
| 00:42 | |-| | taupehat [me@taupehat.com] has quit [Remote host closed the connection] |
| 00:42 | |-| | taupehat [me@taupehat.com] has joined #linode |
| 00:43 | |-| | Zymurgy [zymurgy@cat.delfax.net] has quit [Remote host closed the connection] |
| 00:43 | |-| | Zymurgy [zymurgy@cat.delfax.net] has joined #linode |
| 00:51 | |-| | D[a]rkbeholder [darkbehold@nmathe02.res.csu.edu.au] has quit [Read error: Connection reset by peer] |
| 00:51 | |-| | darkbeholder [darkbehold@nmathe02.res.csu.edu.au] has joined #linode |
| 00:54 | |-| | Marcel [~marcel@jharkema.xs4all.nl] has joined #linode |
| 00:56 | |-| | spr [~spr@ramona.cs.byu.edu] has quit [Quit: Spoon!] |
| 01:12 | |-| | Newsome [~sorenson@ppp-69-218-176-246.dsl.emhril.ameritech.net] has quit [Quit: Linux: Now with employee pricing!] |
| 01:16 | |-| | avid [~x@ca-redbch-cuda1-c2c-166.stmnca.adelphia.net] has joined #linode |
| 01:18 | <jeremiah> | my host server is down!!!. |
| 01:22 | <jeremiah> | anywone else have nodes down? |
| 01:26 | <@caker> | jeremiah: host16 is fine |
| 01:26 | <@mikegrb> | no |
| 01:26 | <@mikegrb> | the host server is just fine |
| 01:26 | <@mikegrb> | # uptime 02:26:46 up 201 days, 12:28, 3 users |
| 01:42 | |-| | jdub [~jdub@ppp121-112.static.internode.on.net] has joined #linode |
| 01:42 | <jdub> | hi all |
| 01:42 | <jdub> | are there uml builds available for off-site testing? |
| 01:42 | <jdub> | same as the ones running on the hosts |
| 01:43 | <jdub> | alternatively, is it relatively sane to run the 2.6.15 kernel? |
| 01:48 | <@mikegrb> | the 2.6 kernels are maked experimental due to lack of nptl support in uml |
| 01:49 | <@mikegrb> | some linode kernels are available here: http://theshore.net/~caker/uml/kernels/ |
| 01:52 | <jeremiah> | thanks mike |
| 01:54 | <jdub> | mikegrb: oh, so that just requires some mangling around with tls directories, right? |
| 01:54 | <jdub> | ahr, thanks for the kernels too |
| 01:55 | <@mikegrb> | yes, mv /lib/tls /lib/tls-disabled, or rm -r /lib/tls, etc |
| 01:55 | [~] | jdub is going to see if dapper runs sanely on 2.4 |
| 01:55 | <@mikegrb> | you should see noticable improvements in preformance with 2.6 vs 2.4 on a linode |
| 01:56 | <jdub> | in 2.6's favour? |
| 01:56 | <@mikegrb> | yes |
| 01:56 | <@mikegrb> | the tls issue precludes us from making it the default though :/ |
| 01:56 | <jdub> | suckage |
| 01:56 | <jdub> | well |
| 01:57 | <jdub> | i guess i should try it out :-) |
| 01:57 | <@caker> | hopefully only until 2.5.17 |
| 01:57 | [~] | jdub was obviously just being a pathetic wuss |
| 01:57 | <@caker> | er, 2.6.17 |
| 01:58 | <@mikegrb> | jdub: well, far better to notice the word experimental and be cautious |
| 01:58 | <@mikegrb> | jdub: too many people just switch without any thought and complaining about everything segfaulting |
| 01:58 | <@mikegrb> | http://www.boingboing.net/2006/03/07/add_intel_drm_to_you.html <-- crazy |
| 01:58 | <jdub> | mikegrb: yeah, totally. |
| 02:02 | <@mikegrb> | caker: http://www.cymru.com/BGP/asnlookup.html check out the references section :> |
| 02:03 | <@caker> | mikegrb: woot :) |
| 02:03 | |-| | MarcelH [~marcel@jharkema.xs4all.nl] has joined #linode |
| 02:03 | |-| | darkbeholder [darkbehold@nmathe02.res.csu.edu.au] has quit [Read error: Connection reset by peer] |
| 02:03 | <@mikegrb> | they also provided some feed back about features they would like to see added |
| 02:03 | <@mikegrb> | which is nice |
| 02:04 | <@mikegrb> | I designed it for use in a project but made it a CPAN module since it would be useful for other things so its good to hear from people with other uses in mind for it |
| 02:04 | |-| | darkbeholder [darkbehold@nmathe02.res.csu.edu.au] has joined #linode |
| 02:08 | |-| | Shaun2222 [~ndci@ip68-5-63-223.oc.oc.cox.net] has joined #linode |
| 02:10 | |-| | Marcel [~marcel@jharkema.xs4all.nl] has quit [Ping timeout: 480 seconds] |
| 02:15 | |-| | Shaun [~ndci@ip68-5-63-223.oc.oc.cox.net] has quit [Ping timeout: 480 seconds] |
| 02:15 | |-| | Shaun2222 changed nick to Shaun |
| 02:50 | <dizzle> | how come the jobs in my queue aren't getting done? |
| 02:51 | <dizzle> | been 30+min |
| 03:00 | [~] | nikki wonders if mikegrb or caker are still around |
| 03:09 | <Internat> | *kicks awstats* |
| 03:32 | <jdub> | mikegrb: i'm now happily running on 2.6 with dapper ;-) thanks! |
| 04:04 | |-| | sneakums [sneakums@jenny.ondioline.org] has joined #linode |
| 04:04 | <sneakums> | caker: mikegrb: my tokens_max is at 1000, did something go haywire or something? |
| 04:13 | <@mikegrb> | hmm |
| 04:13 | <@mikegrb> | swap looks fine |
| 04:13 | <@mikegrb> | and you are at token max so it was probably updatedb or some other cron job |
| 04:14 | <sneakums> | i was seeding a few torrents, but maximum premitted upload oerall was 200KB/s |
| 04:14 | <@mikegrb> | ahh |
| 04:15 | <sneakums> | i did recentlt change swappiness to 30 from 25, maybe it decided to start paging |
| 04:15 | <sneakums> | i've seeded that amount of stuff before without a problem though |
| 04:15 | <@mikegrb> | did you already have it on the linode first? |
| 04:16 | <sneakums> | i torrented it yesterday sometime |
| 04:16 | <sneakums> | then rsynced it off and left the downloaders running |
| 04:16 | <@mikegrb> | I don't remember what the io_rate was running |
| 04:16 | <sneakums> | bt has gone mental on me before, maybe it was simply that |
| 04:16 | <@mikegrb> | it wasn't too bad as you got a token max of 1000 |
| 04:17 | <sneakums> | except with the elevated swappiness it just paged a lot instead of running the kernel out of pages |
| 04:17 | <@mikegrb> | if it had really been causing problems, it would have been in the low hundreds with a refill of 25 or so |
| 04:23 | <jdub> | hey sneakums |
| 04:23 | <sneakums> | hi jdub |
| 04:24 | <jdub> | how've you been? |
| 04:24 | <sneakums> | pretty good |
| 04:24 | <sneakums> | how goes world domination? |
| 04:24 | <jdub> | my cheeks are still pink from the beatings |
| 04:24 | <jdub> | so pretty good overall |
| 04:28 | |-| | Dreamer3 [~dreamer3@0-1pool106-43.nas33.chicago3.il.us.da.qwest.net] has quit [Ping timeout: 480 seconds] |
| 04:31 | |-| | Dreamer3 [~dreamer3@0-1pool107-118.nas33.chicago3.il.us.da.qwest.net] has joined #linode |
| 04:39 | <@mikegrb> | This Message was undeliverable due to the following reason: The user(s) account is temporarily over quota. <postmaster@monornet.hu> |
| 04:39 | <@mikegrb> | Please reply to <Postmaster@monornet.hu> |
| 04:39 | <@mikegrb> | if you feel this message to be in error. |
| 04:39 | <@mikegrb> | awesome |
| 04:41 | <@mikegrb> | lolz |
| 04:41 | <Internat> | lol |
| 04:42 | <Internat> | ive had worse |
| 05:32 | |-| | jekil [~alessandr@europa48.univ.trieste.it] has joined #linode |
| 06:29 | |-| | nybble [~nybble@d150-157-11.home.cgocable.net] has joined #linode |
| 06:32 | |-| | linville [~linville@azure.tuxdriver.com] has joined #linode |
| 06:52 | |-| | nybble [~nybble@d150-157-11.home.cgocable.net] has quit [Quit: Leaving] |
| 07:33 | |-| | Ciaran [~ciaran@host86-132-100-194.range86-132.btcentralplus.com] has joined #linode |
| 07:33 | <Ciaran> | Hi. |
| 07:33 | <Ciaran> | Did host36 just go down? |
| 07:34 | <@mikegrb> | yes |
| 07:34 | <@mikegrb> | linodes are comming back up now |
| 07:35 | <Ciaran> | Okay. |
| 07:35 | <Ciaran> | What was the problem? |
| 07:35 | <@mikegrb> | appears to have become iobound and then lost responsiveness |
| 07:36 | <Ciaran> | Ah, right. |
| 07:38 | <Ciaran> | How long does it generally take for Linodes to start booting? |
| 07:38 | <Ciaran> | I'm not sure - I haven't really been there live as it happened before. |
| 07:39 | <@mikegrb> | I'm not sure about the delay one is started every 30 - 60 seconds or so |
| 07:40 | <Ciaran> | Okay then. |
| 07:41 | <@mikegrb> | what is your username? |
| 07:41 | <Ciaran> | Ciaran. :) |
| 07:42 | <Ciaran> | It's not too much of a problem - it's not urgent that my box is rebooted right now. I was just curious. |
| 07:44 | <@mikegrb> | Mar 8 08:42:56 host36 worker-783[11099]: JobID 233621 finished: success: 1 MSG: |
| 07:44 | <@mikegrb> | ^ you |
| 07:44 | <@mikegrb> | but that job is essentially pressing the power button |
| 07:44 | <@mikegrb> | linode itself still has boot up to go through |
| 07:45 | <Ciaran> | Right. |
| 07:45 | <Ciaran> | Okay, it's up. :D Thanks. |
| 07:57 | |-| | jekil [~alessandr@europa48.univ.trieste.it] has quit [Quit: Leaving] |
| 08:12 | |-| | Redgore [~Redgore@195.38.75.59] has joined #linode |
| 08:44 | |-| | bkirkbri [~Brian@adsl-68-250-65-182.dsl.emhril.ameritech.net] has joined #linode |
| 08:45 | <bkirkbri> | caker or mikegrb? |
| 08:45 | <sneakums> | can't i have both? |
| 08:45 | <bkirkbri> | ha |
| 08:52 | |-| | Jeremy [~jeremy@S0106002078103a54.ek.shawcable.net] has joined #linode |
| 09:00 | <bkirkbri> | mikegrb: around? |
| 09:02 | <bendy24> | not yet |
| 09:02 | <bendy24> | er, i lied |
| 09:02 | <bendy24> | he was last seen an hour and a half ago |
| 09:02 | <bendy24> | probably went to bed |
| 09:02 | <bkirkbri> | bendy24: just got a reply to a ticket, figured we would check in here |
| 09:03 | <bkirkbri> | s/we/he/ |
| 09:03 | [~] | bendy24 shrugs |
| 09:03 | <kvandivo> | he replies to tickets via IP over carrier pigeon |
| 09:05 | <bkirkbri> | IPv4 or IPv6 pigeons? |
| 09:06 | <kvandivo> | not sure.. i know he was on 4, but he had talked about making the hop |
| 09:06 | <bkirkbri> | I hear the IPv6 one have a terrible time finding their way |
| 09:06 | <kvandivo> | it definitely requires a smarter class of fowl |
| 09:07 | <kvandivo> | but, this is linode you're talking about. they've got that covered, right? |
| 09:08 | <bkirkbri> | I can't complain |
| 09:10 | <bendy24> | doesnt the ipv6 pigeon just ride on top of the ipv4 pigeon anyway? |
| 09:10 | <kvandivo> | haven't investigated that thoroughly.. you could be right |
| 09:17 | <jdub> | hrm, host42 having issues, or is it just my uml? |
| 09:36 | |-| | spr [~spr@ramona.cs.byu.edu] has joined #linode |
| 09:58 | <bkirkbri> | jdub: what issues are you having? |
| 10:10 | <bkirkbri> | anyone familiar with the intricacies of the IO limiter? |
| 10:10 | <bkirkbri> | I've never hit it before... |
| 10:10 | <kvandivo> | you can go as far as looking at the kernel patches and figure it out to whatever level of detail you wish |
| 10:11 | <kvandivo> | http://www.linode.com/wiki/index.php/IO_Tokens that might give you the level of detail you want, though.. |
| 10:11 | <bkirkbri> | I might do that, do you have a link? |
| 10:11 | <bkirkbri> | Cool, thanks! |
| 10:13 | |-| | schultmc [~schultmc@zealot.progeny.com] has quit [Quit: Client exiting] |
| 10:16 | |-| | schultmc [~schultmc@zealot.progeny.com] has joined #linode |
| 10:17 | |-| | schultmc [~schultmc@zealot.progeny.com] has quit [Quit: ] |
| 10:17 | <bkirkbri> | Looks like my problem is outside of the kernel limiter, it doesn't have any policy code for adjusting token_refill or token_max itself |
| 10:21 | <sneakums> | i believe 512/400000 are the linode defaults |
| 10:31 | |-| | Bits-n-Pieces [~BnP@c-d28fe055.166-1-64736c10.cust.bredbandsbolaget.se] has quit [Ping timeout: 480 seconds] |
| 10:45 | |-| | Electric1lf changed nick to ElectricElf |
| 10:50 | <linbot> | New news from forums: Host 40 in Sales Questions and Answers <http://www.linode.com/forums/viewtopic.php?t=2132>; |
| 10:51 | |-| | besonen [~besonen@dsl-db.pacinfo.com] has quit [Quit: Leaving] |
| 11:07 | |-| | besonen [~besonen@dsl-db.pacinfo.com] has joined #linode |
| 11:11 | |-| | besonen [~besonen@dsl-db.pacinfo.com] has quit [Read error: Connection reset by peer] |
| 11:11 | |-| | besonen_ [~besonen@dsl-db.pacinfo.com] has joined #linode |
| 11:16 | <bkirkbri> | sneakums: That's what I had until ~4am |
| 11:17 | <bkirkbri> | Now it's 75/500 and has been for 6 hours |
| 11:17 | <Spads> | 500 MAX?? |
| 11:17 | <bkirkbri> | yeah! |
| 11:17 | <kvandivo> | that means you've been thumped.. the general thing that causes that is biting into swap |
| 11:18 | <Spads> | yeah |
| 11:18 | <kvandivo> | 99.9999994% of the time, swap is the culprit |
| 11:18 | <bkirkbri> | Right, I assumed that something went batshit and they had to limit me |
| 11:18 | <Spads> | yeah |
| 11:18 | <bkirkbri> | But I've been digging through logs and munin all morning, the only thing that I see is that the tokens_max went down and then everything started hitting the limiter |
| 11:18 | <Spads> | can you run top to see what it might be? |
| 11:19 | <Spads> | odd |
| 11:19 | <bkirkbri> | I've been tailing logs and watching vmstat all morning |
| 11:19 | <bkirkbri> | There is no swapping and wasn't any at the time of limiting |
| 11:19 | <bkirkbri> | Munin shows no heavy IO or swapping |
| 11:20 | <bkirkbri> | I looked for lots of incoming mail, heavy www traffic... nothing |
| 11:20 | <Spads> | hmmm |
| 11:20 | <kvandivo> | did you file a ticket? |
| 11:20 | <bkirkbri> | I'd like to know what it was, but at this point I'd really like to have >500 tokens_max |
| 11:20 | <Spads> | yeah |
| 11:20 | <bkirkbri> | kvandivo: Yes, and got a few responses from Mike... mostly the usually chiding to tune my linode |
| 11:21 | <bkirkbri> | Which I expect, that's par for the course |
| 11:21 | <bkirkbri> | But the more I look into it, the less I see that would cause limiting |
| 11:21 | <kvandivo> | if he said it, i expect that you were into swap, then |
| 11:21 | <bkirkbri> | I had 12MB swap (but that's stale and hasn't been read/written for ages) |
| 11:22 | <bkirkbri> | Mike misread it as 120MB in a response to my ticket, so... |
| 11:22 | <Spads> | hmmm |
| 11:22 | <bkirkbri> | I run a VERY tight ship with this server, so this is funny |
| 11:22 | <Spads> | Man, I can't wait for Xen |
| 11:23 | <bkirkbri> | Ditto |
| 11:23 | <bkirkbri> | I was on Xen at unixshell for 6 months before Linode |
| 11:23 | <bkirkbri> | Performance was SO much better, but the service/support was nothing like it is here |
| 11:23 | <bkirkbri> | These guys are the tops |
| 11:24 | <bkirkbri> | Anybody know how long it takes for token_max to move back up to a reasonable level? |
| 11:25 | <bkirkbri> | After looking at the caker's kernel patch for the limiter, it looks like adjusting token_max is done by an admin rather than by some heuristic |
| 11:25 | <@mikegrb> | admin or a daemon on the host |
| 11:25 | <taupehat> | moin mikegrb |
| 11:26 | <bkirkbri> | hello |
| 11:26 | <bendy24> | mikegrb: make me an admin |
| 11:26 | <bkirkbri> | That was my other thought ;) |
| 11:26 | <bendy24> | you can all feel safe if i have root |
| 11:26 | <taupehat> | O RLY? |
| 11:27 | <Spads> | QUITE RLY |
| 11:28 | <@mikegrb> | bkirkbri: what's the problem? is a 0.16 loadavg too high? |
| 11:29 | <bkirkbri> | mikegrb: no, not at all... but the occassional 4.32 is :) |
| 11:29 | <taupehat> | oof |
| 11:30 | <bkirkbri> | with refill=75 and max=500 even normal apache/email activity is hitting the limit |
| 11:30 | <@mikegrb> | normal activity shouldn't hit that |
| 11:30 | <bkirkbri> | shouldn't hit 500? |
| 11:30 | <bkirkbri> | i'm confused |
| 11:30 | <@mikegrb> | that part doesn't really matter |
| 11:30 | <Spads> | did the units change? |
| 11:30 | <@mikegrb> | the refill is what really matters |
| 11:31 | <@mikegrb> | the average io rate is 10 |
| 11:31 | <bkirkbri> | hmmm |
| 11:31 | <taupehat> | speaking of which |
| 11:31 | <@mikegrb> | to have a problem with those numbers you have to be doing > a rate of 75 |
| 11:31 | <@mikegrb> | which is 7.5 times normal |
| 11:31 | <taupehat> | mikegrb: My host has been behaving well since I got mysql back on it, yes? |
| 11:31 | <bkirkbri> | Why is the default 512? |
| 11:32 | <taupehat> | bkirkbri: to allow for bursts presumably |
| 11:32 | <@mikegrb> | the default doesn't catch many problems and I don't think it was meant to |
| 11:32 | <@mikegrb> | if the default was meant to catch problems I imagine it would be closer to 100/2000 |
| 11:33 | <bkirkbri> | Right, well bursts is my problem... I sit at tokens=500 (=max) for several minutes, but then a bit of activity causes rate > 75 and it limits |
| 11:33 | <bkirkbri> | but if max=5000 those spikes wouldn't limit |
| 11:33 | <@mikegrb> | the problem is during that bit of activity it hurts all the linodes on your host |
| 11:33 | <bkirkbri> | if I understand correctly |
| 11:34 | <taupehat> | hmm |
| 11:34 | <bkirkbri> | well, I definitely understand the reason for the limiter and don't want to abuse it... I benefit from not having the other people on my host abusing it :) |
| 11:34 | <taupehat> | mikegrb: what kind of IO activity does someone downloading a largish JPG file off a linode generate? |
| 11:34 | <@mikegrb> | there are currently four open tickets from people on your host complaining about preformance |
| 11:35 | <@mikegrb> | taupehat: 1 |
| 11:35 | <bkirkbri> | from host49? |
| 11:35 | <@mikegrb> | yes |
| 11:35 | <bkirkbri> | I had something in the forum just this week about IO performance being dog slow on host49 |
| 11:35 | <bkirkbri> | even a simple "ls -l" took a second or so |
| 11:36 | <bkirkbri> | everything was IOwait |
| 11:36 | <bkirkbri> | but I wasn't near the limit or swapping |
| 11:36 | <taupehat> | so io_tokens are just a count of opened file descriptors then? |
| 11:36 | [~] | taupehat is still on the "downloading the JPG" question |
| 11:36 | <Spads> | I thought it was blocks transferred |
| 11:36 | <@mikegrb> | as I've already stated, the default values don't catch problems |
| 11:37 | <bkirkbri> | right, ok |
| 11:37 | <@mikegrb> | Spads: close, io operations which could be several blocks |
| 11:37 | <Spads> | aha |
| 11:37 | <Spads> | so almost # of seeks |
| 11:37 | <@mikegrb> | yeah |
| 11:37 | <taupehat> | k |
| 11:39 | <Spads> | bkirkbri: do you perhaps have your filesystem mounted sync or something funny like that? |
| 11:39 | <@mikegrb> | anyone see packet loss of nearly 50% just over an hour ago and lasting less then a minute to tp? |
| 11:39 | <Spads> | wasn't looking |
| 11:39 | <bkirkbri> | i had trouble reaching linode.com at that time |
| 11:39 | <bkirkbri> | Spads: nope, just checked |
| 11:40 | <@mikegrb> | bkirkbri: you've said here and in the ticket you have been hitting it with those values.... but since /I/ haven't seen that happen.... |
| 11:40 | <@mikegrb> | I'm going to raise them |
| 11:40 | <@mikegrb> | not all the way back to normal but quite a bit |
| 11:40 | <bkirkbri> | mikegrb: thanks, I'm watching things here very closely |
| 11:41 | <bkirkbri> | Do you have any statistics about what caused you to limit mine in the first place? |
| 11:41 | <bkirkbri> | From logs/munin/etc I don't see heavy activity at that time or any swapping |
| 11:41 | <@mikegrb> | unforuntately not, the process literally generates 10-20k lines of output in the terminal |
| 11:42 | [~] | taupehat is seeing spikes of 15 io_tokens |
| 11:42 | <bkirkbri> | As you said I was 12MB into swap, but that's been there since a big mysqldump a long time ago and wasn't going in/out this morning |
| 11:42 | <@mikegrb> | so the scroll back buffer won't have anything |
| 11:42 | <bkirkbri> | right, well... i'll keep an eye out |
| 11:42 | <@mikegrb> | bkirkbri: which kernel are you using? |
| 11:42 | <kvandivo> | 2.0.28 |
| 11:42 | <@mikegrb> | k good |
| 11:42 | <taupehat> | ! |
| 11:42 | <@mikegrb> | that's our best |
| 11:42 | <bkirkbri> | Linux deeperbydesign.com 2.6.12.3-linode14 #1 Tue Jul 26 23:44:29 EDT 2005 i686 GNU/Linux |
| 11:42 | [~] | mikegrb runs |
| 11:42 | <bendy24> | kvandivo: you'll be stable as a rock! |
| 11:42 | <@mikegrb> | bkirkbri: "echo 1 > /proc/sys/vm/laptop_mode" |
| 11:43 | <bkirkbri> | mikegrb: what does that do? |
| 11:43 | <@mikegrb> | kvandivo: I had a box running 2.2 until not long ago |
| 11:43 | <@mikegrb> | bkirkbri: it tells the kernel vm subsystem to optimize for laptop conditions |
| 11:43 | <kvandivo> | mikegrb: 6 years of uptime? |
| 11:43 | <Spads> | oh yeah |
| 11:43 | <@mikegrb> | bkirkbri: which means minimize disk io to make the battery last as long as possible |
| 11:43 | <Spads> | bkirkbri: I use laptop_mode, and it's great |
| 11:43 | <bendy24> | thats a big updike |
| 11:44 | <Spads> | bendy24: it would overflow the irc buffer |
| 11:44 | <bendy24> | indeed |
| 11:44 | <bkirkbri> | sweet, i googled it but not seeing anything |
| 11:44 | <bkirkbri> | that might help a good bit |
| 11:44 | <bkirkbri> | I thought it swapped out a bit prematurely given my low mem usage |
| 11:44 | <Spads> | [nick@frotz(~)][nick@frotz(~)] grep -v ^# /etc/sysctl.conf |
| 11:44 | <Spads> | vm.laptop_mode=1 |
| 11:44 | <Spads> | ^-- very helpful |
| 11:44 | <Spads> | matter of fact, it should probably be default in the distros |
| 11:45 | <@mikegrb> | bkirkbri: normally it causes less swap usage, in your case it may swap out more (old stuff) in favor of of caching more disk stuffs in ram |
| 11:45 | <bkirkbri> | mikegrb: gotcha |
| 11:45 | <Spads> | I used to have this, which helped a little: |
| 11:45 | <Spads> | #vm.swappiness=25 |
| 11:45 | <Spads> | but the semantics of swappiness were in flux at the time |
| 11:48 | <bkirkbri> | mikegrb: Did you get a warning about my node thrashing this morning or was it just tickets from people on host49? |
| 11:48 | <bkirkbri> | I'm trying to track down what happened so that I can prevent it from happening again |
| 11:48 | <@mikegrb> | tickets |
| 11:49 | <@mikegrb> | your node along would have been fine |
| 11:49 | <@mikegrb> | but your node and another were trying to do around 100-200 io iops/sec |
| 11:49 | <@mikegrb> | and never noncontiguous |
| 11:49 | <bkirkbri> | noncontiguous? like bursts? |
| 11:50 | <@mikegrb> | all over the disk physically |
| 11:50 | <@mikegrb> | cause lots of seeks |
| 11:50 | <bkirkbri> | rsync as backup |
| 11:50 | <@mikegrb> | but that wasn't necessarily your linode specifically, its the two battling for stuff on opposite sides of the disk |
| 11:50 | <@mikegrb> | there is raid mirroring which doubles read throughput |
| 11:50 | <@mikegrb> | but then there are still writes to do |
| 11:51 | <@mikegrb> | and all the linodes on the host |
| 11:51 | <bkirkbri> | Right |
| 11:51 | <bkirkbri> | My node is generally doing many more reads than writes |
| 11:52 | <@mikegrb> | right, most are |
| 11:52 | <bkirkbri> | Munin probably contributes a good bit to my writes and IO overall |
| 11:52 | <bkirkbri> | I'm always tempted to turn it off, but it's really nice to have in times like this one |
| 11:53 | <bkirkbri> | As a postmortem of sorts |
| 11:53 | <bkirkbri> | Any news on Xen timeframe? Last I heard you were waiting on rackspace... |
| 11:53 | <@mikegrb> | yes |
| 11:53 | <@mikegrb> | I think we are compromising on that |
| 11:53 | <@mikegrb> | we wanted giant cages |
| 11:53 | <@mikegrb> | but they didn't even want to sell us small ones |
| 11:54 | <@mikegrb> | and HE kept saying the new datacenter building will be open next week for sure this time |
| 11:54 | <@mikegrb> | so I think the plan is just get another rack at each for now |
| 11:54 | <bkirkbri> | Xenode is going into HE? |
| 11:54 | <@mikegrb> | will be both |
| 11:54 | <@mikegrb> | HE is building a new building in fremont |
| 11:54 | <bkirkbri> | Cool, might get two for redundancy then |
| 11:55 | <@mikegrb> | but we were going to get a cage at HE and a cage at TP |
| 11:55 | <@mikegrb> | neither one has had any free space in 6 months |
| 11:55 | <bkirkbri> | wow |
| 11:56 | <@mikegrb> | someone was using a cage right next to the entrance to the many server room at TP as an office |
| 11:56 | <@mikegrb> | had a desk, chair computer, crt, phone etc |
| 11:56 | <@mikegrb> | even a filing cabinet |
| 11:56 | <@mikegrb> | then one or two racks in the back |
| 11:57 | <@mikegrb> | with that kind of office you wouldn't have to worry about bandwidth |
| 11:57 | <@mikegrb> | gigabit ethernet to your desk |
| 11:57 | <bkirkbri> | that's what i need |
| 11:58 | <kvandivo> | you don't have that? |
| 11:58 | <kvandivo> | bummer |
| 11:58 | <@mikegrb> | kvandivo: shut it, bastard |
| 11:58 | <bkirkbri> | Yeah, stupid netflix... I should have spent that $19 on a gigabit line into my house |
| 11:58 | <@mikegrb> | I just upgraded yesterday |
| 11:59 | <@mikegrb> | 9mbit down/1 mbit up |
| 11:59 | <bkirkbri> | mikegrb: I just ran a rsync to my backup server and it hit an io_rate > 1000 |
| 11:59 | <bkirkbri> | So does that mean that rsyncing things off the machine for backup falls outside the "normal" use of a Linode? |
| 11:59 | <@mikegrb> | no, that is acceptable |
| 12:00 | <@mikegrb> | how long did it last? that's the big thing |
| 12:00 | <bkirkbri> | 10 seconds or so |
| 12:00 | <@mikegrb> | that's no problem then |
| 12:00 | <bkirkbri> | It obviously wasn't 1000 the whole 10 seconds |
| 12:00 | <bkirkbri> | Good, just trying to wrap my head around the numbers |
| 12:01 | <@mikegrb> | the shorter something lasts the higher the max acceptable value is |
| 12:01 | <@mikegrb> | that's why there is a max and a refill |
| 12:01 | <bkirkbri> | So I guess what happened is that you got tickets from people like me that don't like a slow host, I happened to be beating the crap out of the host with a backup at the time and thus the limiting? |
| 12:01 | <@mikegrb> | rather then just a rate |
| 12:02 | <@mikegrb> | that is quite likely |
| 12:02 | <bkirkbri> | Of course, that's why I was saying earlier that >75 was not that bad since it's a spike |
| 12:02 | <bkirkbri> | gotcha |
| 12:02 | <@mikegrb> | does the time you see from your graphs coincide with the backup? |
| 12:02 | <bkirkbri> | Yes |
| 12:02 | <bkirkbri> | Close |
| 12:03 | <bkirkbri> | Plus there is probably a dump of a mysql database sometime around then too |
| 12:03 | <bkirkbri> | Given the interaction of refill and max, I'd probably have been ok if it was 75/2000 but the 75/500 was killing my response time for www clients |
| 12:04 | |-| | Mariano [~Mariano@201.255.104.44] has quit [Quit: Mariano] |
| 12:04 | <Spads> | I think it would be interesting to have a button that blanket set everyone to like 100/2000 for a while to see who actually runs out |
| 12:05 | <bkirkbri> | I appreciate the open discussion. I'm sure you get a lot of whining about IO limits... but I really do obsess over tuning my node so I was suprised |
| 12:05 | <@mikegrb> | Spads: almost nobody |
| 12:05 | <bkirkbri> | If I was still at my old host I'd get a blank "Ticket resolved" email a couple of days late |
| 12:06 | <Spads> | mikegrb: but in a thrash situation, it would be a good test |
| 12:06 | <@mikegrb> | nah |
| 12:06 | <@mikegrb> | we have better tests |
| 12:06 | <Spads> | aha |
| 12:06 | <@mikegrb> | a tool like top that shows everyones tokens |
| 12:06 | <Spads> | that's cool |
| 12:06 | <kvandivo> | a tool tantamount to top? |
| 12:06 | <@mikegrb> | and another (slower) version that shows swap consumption, uptime, and average rate since boot |
| 12:06 | <@mikegrb> | well more like ps |
| 12:07 | <@mikegrb> | as it's a one shot thing |
| 12:07 | <@mikegrb> | but sorted and such |
| 12:07 | <bkirkbri> | I think that my IO rate average since my last reboot 30 days ago is around 20 |
| 12:08 | <@mikegrb> | 23 |
| 12:08 | <@mikegrb> | good guess |
| 12:08 | <kvandivo> | i'm at 8.3 |
| 12:08 | <@mikegrb> | about 2x average but not too bad |
| 12:08 | <@mikegrb> | we start getting mean around 50 or so |
| 12:08 | <Spads> | what's my average? |
| 12:09 | <@mikegrb> | if your average had been 100+ you would have gotten 10/300 or some such |
| 12:09 | <@mikegrb> | iocheck zork |
| 12:09 | <@mikegrb> | er |
| 12:09 | <Spads> | haha |
| 12:10 | <bkirkbri> | I think I'll move munin to my backup host and leave just the node running on the linode |
| 12:10 | <@mikegrb> | jesus |
| 12:10 | <@mikegrb> | 63 |
| 12:10 | <Spads> | ouch |
| 12:11 | <bkirkbri> | Spads: You shouldn't have asked |
| 12:11 | <Spads> | haha |
| 12:11 | <Spads> | over what period? |
| 12:11 | <kvandivo> | since reboot |
| 12:11 | <Spads> | heh |
| 12:11 | <Spads> | I've got a user with a large mutt maildir |
| 12:11 | <@mikegrb> | even linode.com is only 40 |
| 12:11 | <Spads> | and for some reason it keeps re-scanning |
| 12:12 | <@mikegrb> | he should archive the other stuff in a sub folder |
| 12:12 | <Spads> | and I think that makes a lot of disk noise |
| 12:12 | <Spads> | yes he should |
| 12:12 | <@mikegrb> | threaten him with a stick |
| 12:12 | [~] | Beirdo hands Spads a LART |
| 12:13 | <Spads> | even better, mutt should fix itself |
| 12:13 | <@mikegrb> | no client handles giant folders |
| 12:13 | <Spads> | we're all doing header caching, but still he ends up with rescans |
| 12:13 | <@mikegrb> | + well |
| 12:15 | |-| | jekil [~alessandr@82.52.168.149] has joined #linode |
| 12:16 | |-| | schultmc [~schultmc@zealot.progeny.com] has joined #linode |
| 12:17 | [~] | mikegrb is in the process of switching news readers |
| 12:17 | <@mikegrb> | I haven't checked but I think the new one will read the old one's killfile |
| 12:17 | <@mikegrb> | which is a must |
| 12:17 | <@mikegrb> | but the interface looks a lot like mutt and I keep hitting mutt commands which do the absolute wrong thing :< |
| 12:18 | <Spads> | slrn? |
| 12:18 | <@mikegrb> | yes |
| 12:18 | <Spads> | yeah |
| 12:18 | <Spads> | I did a lot of remapping for slrn |
| 12:18 | <Spads> | but the interface is just different enough |
| 12:19 | <@mikegrb> | I think the lead developer dude uses emacs |
| 12:19 | <Spads> | heh |
| 12:19 | <@mikegrb> | it's all ESC this ESC that |
| 12:19 | <Spads> | yeah |
| 12:19 | <Spads> | you can use alt |
| 12:20 | <@mikegrb> | and it won't convert the date header to my timezone, by default anyway |
| 12:20 | <@mikegrb> | I don't have mutt doing that either, but at least with email I can hit h and see the header my mail server added with a received line in my timezone |
| 12:23 | |-| | Bits-n-Pieces [~BnP@c-d28fe055.166-1-64736c10.cust.bredbandsbolaget.se] has joined #linode |
| 12:31 | <Spads> | okay, we're blowing away our header caches |
| 12:31 | <Spads> | in the hopes that it'll all WORK this time |
| 12:34 | <bkirkbri> | mikegrb: Thanks for all the help, I'm out |
| 12:34 | |-| | bkirkbri [~Brian@adsl-68-250-65-182.dsl.emhril.ameritech.net] has left #linode [] |
| 12:53 | <jdub> | mikegrb: thanks for the ticket - i've been watching the host, trying to determine what the issue is |
| 12:54 | <@mikegrb> | no problem |
| 12:54 | <jdub> | suffer me for upgrading ;) |
| 12:54 | <@mikegrb> | heh |
| 13:58 | <gpd> | pop quiz: get find to display on the filename (no path) |
| 13:59 | <gpd> | s/on/only/ |
| 14:07 | <@mikegrb> | find -print | sed |
| 14:07 | [~] | mikegrb runs |
| 14:07 | <gpd> | find . | sed 's/.*\///' -- heh |
| 14:07 | <gpd> | what I concluded also |
| 14:09 | <@mikegrb> | perhaps find can do it |
| 14:09 | <@mikegrb> | sometimes it's quicker just to pipe it to something else |
| 14:09 | <gpd> | word |
| 15:01 | |-| | jekil [~alessandr@82.52.168.149] has quit [Read error: Connection reset by peer] |
| 15:29 | |-| | kic [~rfoote@70.158.34.157] has joined #linode |
| 15:29 | <kic> | hola |
| 15:29 | <kic> | I searched the forums for instances of 'Unable to determine CPU Frequency' when installing some software.. but can't find anyone else mentioning it |
| 15:29 | <kic> | anyone know anything about this error? |
| 15:38 | <gpd> | steam? |
| 15:38 | <kic> | yeah |
| 15:39 | <kic> | hrm, seems to have disappeard now though |
| 15:39 | <kic> | its installing :D |
| 15:39 | <kic> | may happen again during runtime though |
| 15:39 | <kic> | from what i've been reading |
| 15:39 | <kic> | have you had experience with that? |
| 15:39 | <gpd> | no - just google'd |
| 15:40 | <kic> | yeah, i did the same |
| 15:40 | <kic> | didn't turn up much :D |
| 15:57 | |-| | MarcelH [~marcel@jharkema.xs4all.nl] has quit [Quit: Leaving] |
| 15:57 | |-| | Marcel [~marcel@jharkema.xs4all.nl] has joined #linode |
| 16:17 | |-| | linville [~linville@azure.tuxdriver.com] has quit [Quit: Leaving] |
| 16:20 | <linbot> | New news from forums: Importing a mysqldump with BLOBs in Linux, Apache, Mysql and PHP (LAMP) Forum <http://www.linode.com/forums/viewtopic.php?t=2123>; |
| 16:29 | |-| | Dreamer3 [~dreamer3@0-1pool107-118.nas33.chicago3.il.us.da.qwest.net] has quit [Ping timeout: 480 seconds] |
| 16:30 | |-| | Dreamer3 [~dreamer3@0-1pool106-100.nas33.chicago3.il.us.da.qwest.net] has joined #linode |
| 16:34 | |-| | kic [~rfoote@70.158.34.157] has quit [Quit: ] |
| 16:39 | |-| | Sgeo [~Sgeo@ool-18bf61f7.dyn.optonline.net] has joined #linode |
| 16:42 | |-| | Redgore [~Redgore@195.38.75.59] has quit [Quit: A geek without purpose - http://martlev.com | SMDC-Network IRC - irc.smdc-network.org] |
| 16:55 | |-| | jekil [~alessandr@host149-168.pool8252.interbusiness.it] has joined #linode |
| 16:57 | |-| | ScytheBlade1 [~Death@c-67-186-244-23.hsd1.ut.comcast.net] has joined #linode |
| 16:58 | [~] | ScytheBlade1 thinks his linode just died |
| 16:58 | <ScytheBlade1> | host36 reads idle |
| 16:58 | <ScytheBlade1> | ... but I have no connectivity from two different IPs |
| 16:58 | [~] | ScytheBlade1 pokes the ops, caker and mikegrb |
| 17:09 | |-| | spr [~spr@ramona.cs.byu.edu] has quit [Quit: Spoon!] |
| 17:12 | <ScytheBlade1> | never mind, it just rebooted and broke hard for no reason... :/ |
| 17:12 | <ScytheBlade1> | Why it rebooted, I know not, but eh |
| 17:12 | <Spads> | ouch |
| 17:12 | <Spads> | yeah |
| 17:31 | <taupehat> | hmm |
| 17:31 | <taupehat> | wasn't there a sticky somewhere in the forums on performance-tuning? |
| 17:31 | <taupehat> | I'm looking at how to trim an apache config on another machine =] |
| 17:37 | <Spads> | I am running teh Xgl |
| 17:37 | <Spads> | it is super unstable on this radeon 7500 |
| 18:18 | <gpd> | Spads: I have compiz on my 6800 - also fairly unstable |
| 18:18 | <gpd> | not sure I like it either - the standard settings make you feel a bit ill after a while |
| 18:18 | <Spads> | my problem is that I get flicker and dropout stuff |
| 18:18 | <gpd> | flicker can be fixed... change the refresh option |
| 18:18 | <Spads> | oh? |
| 18:18 | <gpd> | let me look |
| 18:19 | <Spads> | I may give it another try |
| 18:19 | <Spads> | but also all my Xv apps are slow as treacle |
| 18:19 | <gpd> | in the dapper ubuntuforums.org there is a thread that mentions it |
| 18:20 | <gpd> | http://ubuntuforums.org/showthread.php?t=131267 |
| 18:22 | <gpd> | Does xgl have shadows - ala OSX? not seen it so far |
| 18:23 | <Spads> | yes |
| 18:23 | <Spads> | gpd: I see no mention of flicker in that link |
| 18:24 | <gpd> | s/flicker/wobble/ |
| 18:24 | <Spads> | oh |
| 18:24 | <gpd> | the bit with the gconf-editor and the refresh_rate |
| 18:24 | <gpd> | worked for me |
| 18:24 | <Spads> | um |
| 18:24 | <Spads> | I see wobble mentioned there, but that's a feature |
| 18:24 | <Spads> | I'm talking about jagged triangular refresh problems |
| 18:25 | <gpd> | when the wobble is in action you get flicker if the refresh is 85 |
| 18:25 | <Spads> | well |
| 18:25 | <Spads> | everything flickers for me |
| 18:25 | <Spads> | *everything* |
| 18:25 | <gpd> | oh |
| 18:25 | <gpd> | not so good then |
| 18:25 | <Spads> | and I also have drop-out sections |
| 18:25 | <gpd> | ubuntu dapper? |
| 18:25 | <Spads> | yes |
| 18:25 | <Spads> | radeon 7500 |
| 18:26 | <gpd> | there are radeon specific tips elsewhere in that forum |
| 18:26 | <Spads> | mobility |
| 18:26 | <Spads> | Most involve fglrc |
| 18:26 | <Spads> | fglrx |
| 18:26 | <Spads> | trust me |
| 18:26 | <Spads> | this is just lack of support for this chipset |
| 18:26 | <gpd> | oh well - lucky me then with my nvidia goodness |
| 18:26 | <Spads> | and your proprietary drivers |
| 18:26 | [~] | gpd cackles |
| 18:27 | <gpd> | I am tainted |
| 18:27 | <Spads> | be sure to bow and scrape extra low the next time X or the kernel changes something and nvidia don't notice |
| 18:27 | <Spads> | maybe you can beg them to bring old drivers up to date |
| 18:27 | <Spads> | I'm sure they'll care. |
| 18:27 | <Spads> | They'll rush right over and say "OUR DRIVERS ARE SUPPORTED WITH RHCE 8.2" |
| 18:27 | <Spads> | etc. |
| 18:27 | <gpd> | I'll get my whipping boys at IBM to talk to them |
| 18:28 | <Spads> | um |
| 18:28 | <Spads> | that's hilarious |
| 18:28 | <Spads> | a whipping boy is a slave you beat when a prince does something wrong |
| 18:29 | <Spads> | not some sort of ambassador |
| 18:38 | |-| | Electric1lf [~dbharris@69.158.86.71] has joined #linode |
| 18:48 | |-| | Electric2lf [~dbharris@69.158.85.49] has joined #linode |
| 18:49 | |-| | ElectricElf [~dbharris@electricelf.noc.oftc.net] has quit [Read error: Connection timed out] |
| 18:49 | <cmantito> | Out of curiosity, are there any lesbian couples currently living within the walls of the TP datacentre? I ask because I got spam saying there were lesbian couples looking for a manfriend in my area. |
| 18:50 | <cmantito> | Seeing as how my area consists of this room, and everything connected by a TCP socket in my brain... |
| 18:50 | <cmantito> | And there are no lesbian couples in here. |
| 18:50 | |-| | Electric1lf [~dbharris@69.158.86.71] has quit [Ping timeout: 480 seconds] |
| 19:03 | <encode> | hahahah cmantito |
| 19:05 | <fo0bar> | cmantito: the lesbians are calling from INSIDE THE HOUSE |
| 19:10 | <cmantito> | =P |
| 19:37 | |-| | spr [~spr@192.41.84.58] has joined #linode |
| 19:42 | |-| | spr [~spr@192.41.84.58] has quit [Remote host closed the connection] |
| 20:03 | |-| | itamarjp [~itamar@200-233-243-063.xd-dynamic.ctbcnetsuper.com.br] has joined #linode |
| 20:19 | |-| | itamarjp [~itamar@200-233-243-063.xd-dynamic.ctbcnetsuper.com.br] has quit [Quit: ] |
| 20:24 | |-| | jdub [~jdub@ppp121-112.static.internode.on.net] has left #linode [] |
| 20:35 | |-| | besonen_ [~besonen@dsl-db.pacinfo.com] has quit [Read error: Connection reset by peer] |
| 20:36 | |-| | besonen_ [~besonen@dsl-db.pacinfo.com] has joined #linode |
| 20:36 | |-| | darkbeholder [darkbehold@nmathe02.res.csu.edu.au] has quit [Read error: Connection reset by peer] |
| 20:37 | |-| | darkbeholder [darkbehold@nmathe02.res.csu.edu.au] has joined #linode |
| 21:04 | |-| | jekil [~alessandr@host149-168.pool8252.interbusiness.it] has quit [Quit: Leaving] |
| 21:32 | |-| | Dreamer3 [~dreamer3@0-1pool106-100.nas33.chicago3.il.us.da.qwest.net] has quit [Ping timeout: 480 seconds] |
| 21:50 | <encode> | fo0bar: that would be a scary thing |
| 22:59 | |-| | VS_ChanLog [~stats@ns.theshore.net] has left #linode [Rotating Logs] |
| 22:59 | |-| | VS_ChanLog [~stats@ns.theshore.net] has joined #linode |
| 23:23 | |-| | Sgeo [~Sgeo@ool-18bf61f7.dyn.optonline.net] has quit [Quit: Konversation terminated!] |
| --- | Log | closed Thu Mar 09 00:00:24 2006 |
