crt0mega
/
skipfish
1
0
Fork 0
Code Issues 1 Pull Requests Projects Releases Wiki Activity

OOM error #3

New Issue
Closed
opened 2017-07-18 07:12:10 +00:00 by youradds · 3 comments
youradds commented 2017-07-18 07:12:10 +00:00 (Migrated from github.com)
Copy Link

Hi,

Is anyone else getting OOM errors? I'm on a pretty decent spec machine, but it just doesn't want to run on a large site! I'm testing it using:

./skipfish -B .google-analytics.com -B .googleapis.com --flush-to-disk -r 800000 -M -L -e -m 5 -g 10 -o output_folder11 http://www.graphicsfactory.com

It runs for a while, and then gets "Killed!". Looking in kern.log, I can see its running out of memory:

Jul 18 06:38:33 admin kernel: skipfish invoked oom-killer: gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, oom_score_adj=0
Jul 18 06:38:33 admin kernel: skipfish cpuset=/ mems_allowed=0
Jul 18 06:38:33 admin kernel: CPU: 0 PID: 9319 Comm: skipfish Not tainted 4.9.15-x86_64-linode81 #1
Jul 18 06:38:33 admin kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.1-0-gb3ef39f-prebuilt.qemu-project.org 04/01/2014
Jul 18 06:38:33 admin kernel: 0000000000000000 ffffffff817012a5 0000000000000000 ffffc90006677c90
Jul 18 06:38:33 admin kernel: ffff880075edcb00 ffffffff8122edc7 0000000000000000 ffff880100000000
Jul 18 06:38:33 admin kernel: ffff88013fc18a20 ffff88012279d780 0000000000000001 ffffffff8102b6b9
Jul 18 06:38:33 admin kernel: Call Trace:
Jul 18 06:38:33 admin kernel: [<ffffffff817012a5>] ? dump_stack+0x5d/0x88
Jul 18 06:38:33 admin kernel: [<ffffffff8122edc7>] ? dump_header+0x7f/0x221
Jul 18 06:38:33 admin kernel: [<ffffffff8102b6b9>] ? __switch_to+0x1f9/0x5c0
Jul 18 06:38:33 admin kernel: [<ffffffff811377ce>] ? pick_next_task_fair+0x37e/0x440
Jul 18 06:38:33 admin kernel: [<ffffffff811d4d5a>] ? oom_kill_process+0x25a/0x440
Jul 18 06:38:33 admin kernel: [<ffffffff811092d5>] ? has_ns_capability_noaudit+0x15/0x20
Jul 18 06:38:33 admin kernel: [<ffffffff811d497c>] ? oom_badness+0x10c/0x180
Jul 18 06:38:33 admin kernel: [<ffffffff811d51f2>] ? out_of_memory+0x112/0x470
Jul 18 06:38:33 admin kernel: [<ffffffff811d9d1f>] ? __alloc_pages_nodemask+0xd2f/0xe80
Jul 18 06:38:33 admin kernel: [<ffffffff816db5af>] ? __blk_run_queue+0x2f/0x40
Jul 18 06:38:33 admin kernel: [<ffffffff8121b1ca>] ? alloc_pages_current+0x9a/0x120
Jul 18 06:38:33 admin kernel: [<ffffffff811d33ab>] ? filemap_fault+0x35b/0x540
Jul 18 06:38:33 admin kernel: [<ffffffff8120d88b>] ? page_add_file_rmap+0x3b/0x60
Jul 18 06:38:33 admin kernel: [<ffffffff8131318f>] ? ext4_filemap_fault+0x3f/0x60
Jul 18 06:38:33 admin kernel: [<ffffffff811fee9c>] ? __do_fault+0x6c/0xd0
Jul 18 06:38:33 admin kernel: [<ffffffff812046c8>] ? handle_mm_fault+0x8b8/0xdf0
Jul 18 06:38:33 admin kernel: [<ffffffff8105a8e5>] ? __do_page_fault+0x195/0x520
Jul 18 06:38:33 admin kernel: [<ffffffff81c2fcf8>] ? async_page_fault+0x28/0x30
Jul 18 06:38:33 admin kernel: Mem-Info:
Jul 18 06:38:33 admin kernel: active_anon:693249 inactive_anon:273191 isolated_anon:0
Jul 18 06:38:33 admin kernel: active_file:158 inactive_file:249 isolated_file:0
Jul 18 06:38:33 admin kernel: unevictable:0 dirty:1 writeback:4 unstable:0
Jul 18 06:38:33 admin kernel: slab_reclaimable:3729 slab_unreclaimable:11567
Jul 18 06:38:33 admin kernel: mapped:5680 shmem:43627 pagetables:6374 bounce:0
Jul 18 06:38:33 admin kernel: free:6892 free_pcp:356 free_cma:0
Jul 18 06:38:33 admin kernel: Node 0 active_anon:2772996kB inactive_anon:1092764kB active_file:632kB inactive_file:996kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:22720kB dirty:4kB writeback:16kB shmem:174508kB writeb$
Jul 18 06:38:33 admin kernel: Node 0 DMA free:15732kB min:28kB low:40kB high:52kB active_anon:160kB inactive_anon:0kB active_file:0kB inactive_file:16kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB slab_r$
Jul 18 06:38:33 admin kernel: lowmem_reserve[]: 0 2968 3927 3927

Am I doing something wrong? It's a large site to test, but 4gb of RAM should be more than enough!

Thanks

Andy

Hi, Is anyone else getting OOM errors? I'm on a pretty decent spec machine, but it just doesn't want to run on a large site! I'm testing it using: > ./skipfish -B .google-analytics.com -B .googleapis.com --flush-to-disk -r 800000 -M -L -e -m 5 -g 10 -o output_folder11 http://www.graphicsfactory.com It runs for a while, and then gets "Killed!". Looking in kern.log, I can see its running out of memory: ``` Jul 18 06:38:33 admin kernel: skipfish invoked oom-killer: gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, oom_score_adj=0 Jul 18 06:38:33 admin kernel: skipfish cpuset=/ mems_allowed=0 Jul 18 06:38:33 admin kernel: CPU: 0 PID: 9319 Comm: skipfish Not tainted 4.9.15-x86_64-linode81 #1 Jul 18 06:38:33 admin kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.1-0-gb3ef39f-prebuilt.qemu-project.org 04/01/2014 Jul 18 06:38:33 admin kernel: 0000000000000000 ffffffff817012a5 0000000000000000 ffffc90006677c90 Jul 18 06:38:33 admin kernel: ffff880075edcb00 ffffffff8122edc7 0000000000000000 ffff880100000000 Jul 18 06:38:33 admin kernel: ffff88013fc18a20 ffff88012279d780 0000000000000001 ffffffff8102b6b9 Jul 18 06:38:33 admin kernel: Call Trace: Jul 18 06:38:33 admin kernel: [<ffffffff817012a5>] ? dump_stack+0x5d/0x88 Jul 18 06:38:33 admin kernel: [<ffffffff8122edc7>] ? dump_header+0x7f/0x221 Jul 18 06:38:33 admin kernel: [<ffffffff8102b6b9>] ? __switch_to+0x1f9/0x5c0 Jul 18 06:38:33 admin kernel: [<ffffffff811377ce>] ? pick_next_task_fair+0x37e/0x440 Jul 18 06:38:33 admin kernel: [<ffffffff811d4d5a>] ? oom_kill_process+0x25a/0x440 Jul 18 06:38:33 admin kernel: [<ffffffff811092d5>] ? has_ns_capability_noaudit+0x15/0x20 Jul 18 06:38:33 admin kernel: [<ffffffff811d497c>] ? oom_badness+0x10c/0x180 Jul 18 06:38:33 admin kernel: [<ffffffff811d51f2>] ? out_of_memory+0x112/0x470 Jul 18 06:38:33 admin kernel: [<ffffffff811d9d1f>] ? __alloc_pages_nodemask+0xd2f/0xe80 Jul 18 06:38:33 admin kernel: [<ffffffff816db5af>] ? __blk_run_queue+0x2f/0x40 Jul 18 06:38:33 admin kernel: [<ffffffff8121b1ca>] ? alloc_pages_current+0x9a/0x120 Jul 18 06:38:33 admin kernel: [<ffffffff811d33ab>] ? filemap_fault+0x35b/0x540 Jul 18 06:38:33 admin kernel: [<ffffffff8120d88b>] ? page_add_file_rmap+0x3b/0x60 Jul 18 06:38:33 admin kernel: [<ffffffff8131318f>] ? ext4_filemap_fault+0x3f/0x60 Jul 18 06:38:33 admin kernel: [<ffffffff811fee9c>] ? __do_fault+0x6c/0xd0 Jul 18 06:38:33 admin kernel: [<ffffffff812046c8>] ? handle_mm_fault+0x8b8/0xdf0 Jul 18 06:38:33 admin kernel: [<ffffffff8105a8e5>] ? __do_page_fault+0x195/0x520 Jul 18 06:38:33 admin kernel: [<ffffffff81c2fcf8>] ? async_page_fault+0x28/0x30 Jul 18 06:38:33 admin kernel: Mem-Info: Jul 18 06:38:33 admin kernel: active_anon:693249 inactive_anon:273191 isolated_anon:0 Jul 18 06:38:33 admin kernel: active_file:158 inactive_file:249 isolated_file:0 Jul 18 06:38:33 admin kernel: unevictable:0 dirty:1 writeback:4 unstable:0 Jul 18 06:38:33 admin kernel: slab_reclaimable:3729 slab_unreclaimable:11567 Jul 18 06:38:33 admin kernel: mapped:5680 shmem:43627 pagetables:6374 bounce:0 Jul 18 06:38:33 admin kernel: free:6892 free_pcp:356 free_cma:0 Jul 18 06:38:33 admin kernel: Node 0 active_anon:2772996kB inactive_anon:1092764kB active_file:632kB inactive_file:996kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:22720kB dirty:4kB writeback:16kB shmem:174508kB writeb$ Jul 18 06:38:33 admin kernel: Node 0 DMA free:15732kB min:28kB low:40kB high:52kB active_anon:160kB inactive_anon:0kB active_file:0kB inactive_file:16kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB slab_r$ Jul 18 06:38:33 admin kernel: lowmem_reserve[]: 0 2968 3927 3927 ``` Am I doing something wrong? It's a large site to test, but 4gb of RAM should be more than enough! Thanks Andy
AlekKras commented 2018-09-17 22:12:23 +00:00 (Migrated from github.com)
Copy Link

Hmm, that looks interesting. I have not had any issues previously with this. @spinkham what do you think about it?

Hmm, that looks interesting. I have not had any issues previously with this. @spinkham what do you think about it?
spinkham commented 2018-09-17 23:13:23 +00:00 (Migrated from github.com)
Copy Link

Not sure: You used the flags I would recommend trying for memory issues.
To be clear though, Skipfish is @lcamtuf's work, mostly trying to keep it available and haven't had my fingers too deep in the guts.

Not sure: You used the flags I would recommend trying for memory issues. To be clear though, Skipfish is @lcamtuf's work, mostly trying to keep it available and haven't had my fingers too deep in the guts.
youradds commented 2018-09-18 06:29:11 +00:00 (Migrated from github.com)
Copy Link

No worries. I ended up going down another route in the end (as this was 2017 I posted =)

No worries. I ended up going down another route in the end (as this was 2017 I posted =)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
master
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: crt0mega/skipfish#3
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?