0005986: Designating large areas for smoothing slows game to a crawl

Notes
(0022846) Footkerchief (manager) 2012-06-04 08:10	Reminder sent to: Leonassan Does this reproduce in every fortress? If not, please upload a save demonstrating the problem to http://dffd.wimbli.com/ [^] and post the link here.

(0022849) Leonassan (reporter) 2012-06-04 13:39	Done. I haven't managed to get a second fort deep enough to replicate, so here's the save. http://dffd.wimbli.com/file.php?id=6423 [^]

(0022850) Leonassan (reporter) 2012-06-04 14:01	It's very apparent where the save currently is. I loaded it up and was at 7FPS, as soon as the dwarves finished their current engrave designation, it jumped to 31.

(0022856) Mopsy (reporter) 2012-06-05 03:21	I've been noticing similar behavior in a large, populous 40d fort. Could it be a consequence of the alleged "dwarves don't seek jobs, jobs seek dwarves" design? Unlike a designation for mining into solid rock, a smoothing designation often creates a large number of simultaneously reachable tasks. If all those tasks go looking for an engraver once every N steps, we might see FPS drop significantly. The FPS drop might also depend on the number of dwarves in the fort: (CPU cycles used for smoothing job assignment) = K * n(reachable smoothing-designated tiles) * n(dwarves to check for availability as engravers).

(0022858) blake77 (reporter) 2012-06-05 05:32	I also noticed this behaviour in previous versions of DF.

(0022859) ag (reporter) 2012-06-05 11:34	Ran this through my profiling setup on Core i5 (got 20 FPS): http://cloud.github.com/downloads/angavrilov/dfprofile/log.slow-engrave.4a.svg [^] Observations: 1) Engraving job creation does take more than 7% CPU; 5 of those is in the loop over all units, including dead. I would guess the function is called for every designated tile. 2) More than 15% CPU is used by searching for items by id - general_ref_item::getItem is obvious, but there are also other instances inlined in various other functions, e.g. 2% in map_block::updateTemp. 3) There are also other item-related functions mentioned, including things like the map renderer, which walks through all items on the map to check if they are visible now. This supports the generally accepted opinion that large number of units (2500+ on this map) and items (50000+) causes lag. Note: in viewing the graph bear in mind that true stack traces are unavailable, so the 'total time spent' percentage in parentheses is computed purely heuristically, and may significantly deviate from the truth.

(0022865) Leonassan (reporter) 2012-06-05 17:31 edited on: 2012-06-05 17:34	Perhaps pruning dead units from the units list will alleviate the problem. I'll try it and see if it helps any. I didn't notice the slowdown until after I breached the caverns, so that could be where the additional units are coming from. Perhaps the loop to check for units for a job should be limited to alive dwarves only? Edit: Also, to check the theory of large multiple accessible jobs, I've found the same behavior with mining designations that can all be reached, and massive dumping designations, so I believe you are correct that it has to do with a lot of tasks that can be completed being queued at the same time. Edit: Ok, I cannot remove dead units from the units list.

(0022867) Quietust (reporter) 2012-06-05 18:13	> Ok, I cannot remove dead units from the units list. Of course you can't - you'd need DFHack to do something like that.

(0022868) Leonassan (reporter) 2012-06-05 18:17	> Of course you can't - you'd need DFHack to do something like that. Or since you can remove units from your alive, active dwarves list, this ability should be added to the Dead/Missing units list as well.

(0022870) ag (reporter) 2012-06-06 02:44 edited on: 2012-06-06 03:20	Some experimental results: - Nuking all items except those in buildings and constructions got fps from 20 to 50. - Further 5 fps recovered by sealing that pump contraption - you should never make areas with changing water level reachable from the rest of the fortress. - What remained is 20% unit temperature update cost, 20% unit enemy/friend checks, 20% pathfinding through all those long and wide areas as units drag plants to stockpiles. Edit: here's the graph: http://cloud.github.com/downloads/angavrilov/dfprofile/log.slow-engrave.5d.svg [^]

(0022877) Leonassan (reporter) 2012-06-06 12:54	I'm sure there are other things that are causing lag, but that doesn't change the fact that smoothing is causing lag :)

Issue History
Date Modified	Username	Field	Change
2012-06-04 03:26	Leonassan	New Issue
2012-06-04 03:26	Leonassan	Issue Monitored: Leonassan
2012-06-04 03:26	Leonassan	Issue End Monitor: Leonassan
2012-06-04 08:10	Footkerchief	Note Added: 0022846
2012-06-04 13:39	Leonassan	Note Added: 0022849
2012-06-04 14:01	Leonassan	Note Added: 0022850
2012-06-05 03:21	Mopsy	Note Added: 0022856
2012-06-05 03:29	Mopsy	Issue Monitored: Mopsy
2012-06-05 05:32	blake77	Note Added: 0022858
2012-06-05 11:34	ag	Note Added: 0022859
2012-06-05 17:31	Leonassan	Note Added: 0022865
2012-06-05 17:32	Leonassan	Note Edited: 0022865	View Revisions
2012-06-05 17:34	Leonassan	Note Edited: 0022865	View Revisions
2012-06-05 18:13	Quietust	Note Added: 0022867
2012-06-05 18:17	Leonassan	Note Added: 0022868
2012-06-06 02:44	ag	Note Added: 0022870
2012-06-06 03:20	ag	Note Edited: 0022870	View Revisions
2012-06-06 12:54	Leonassan	Note Added: 0022877
2012-06-07 01:45	Buglist	Issue Monitored: Buglist
2013-12-07 04:30	Thundercraft	Issue Monitored: Thundercraft

Relationships