0010459: Game stuttering/freezing every couple of steps

Notes
(0037346) lethosor (manager) 2017-12-17 15:14	See also https://www.reddit.com/r/dwarffortress/comments/7kal8b/biweekly_df_questions_thread/drd3orx/ [^] - doesn't seem utility-dependent.

(0037350) Sarmatian123 (reporter) 2017-12-18 14:34 edited on: 2017-12-18 14:39	Reproducible : Not always random. It seems I always construct fortress in 5-10 years with 20k+ blocks, so my fps drops even with 100 Dwarves to 15(15) fps. So what my staggering low fps have to do with finding some game mechanics causing such staggering freezes? Well, I had entire games turning into staggering frames, as well as staggering, which refuse to let be for entire season, or short periods of such stagger. They were out of blue sky to me. However at few occasions I noticed some particular events taking place for entire duration of staggering. This event is combat. Maybe my game did not progress much. Lagged. Staggered. Froze even. However one thing grew very fast and with lots of pages. This was combat report. There were like 20+ pages over time long like few seconds for many characters involved. I can not be 100% sure on the cause of all fps stagger. For sure combat at low fps, can sometimes cause it. Also thing to keep in mind, even if we don't dig into caverns/lava/hell, there are still animals spawning there and pathing could cause low fps too.

(0037354) sionlife (reporter) 2017-12-19 06:21 edited on: 2017-12-19 06:40	looks like duplicate of http://www.bay12games.com/dwarves/mantisbt/view.php?id=8076 [^] http://www.bay12games.com/dwarves/mantisbt/view.php?id=9109 [^] As a note, I have also experienced this stutter phenomenon on forts in 44.02 that have reached 200 populations. Not as bad as the OP but I get a stutter of about 0.5-1s every 5s or so. It is very noticeable. I don't remember having quite so many problems on 43.05.

(0037363) sionlife (reporter) 2017-12-20 08:59	I have found that if you retire the fortress and then reclaim it back, it cures the stuttering problem. You will lose all your drinks though.

(0037370) lethosor (manager) 2017-12-21 19:24	sionlife: it's hard to tell for sure if those are duplicates, since their causes are unclear. I've marked this as related to 0009109. The save in 0008076 has disappeared, so I closed it for now.

(0037371) lethosor (manager) 2017-12-21 19:25	Reminder sent to: Sarmatian123, Suspriscus Can you check on the size of your errorlog.txt and gamelog.txt files? Are they significantly increasing when stuttering happens?

(0037373) Sarmatian123 (reporter) 2017-12-22 04:09 edited on: 2017-12-22 04:34	I never check those files, unless something crashes. DF is very stable. It crashed for me only recently on 0.43.05 on 2 occasions. One was directly after starting new embark. I just continued from save and it was magically fine then. Second was after I was messing with different fertilization settings (seasonal vs fertilization) for growing crops on a field. This is not on the topic though... Sorry! I do check however the "a" and "r" menus. This is how I discovered during couple of those staggering periods some noticeable increases in combat reports. So no, size of those files are increasing only with how long I played given DF version with one exception. 0.42.04 errorlog.txt is significantly larger (around 300kB), then entire 0.43.05 (around 200kb), which I played for significantly longer time with more embarks, each lasting longer as well. It is also the version I was experiencing the significant 10 sec freeze, 10 sec stagger slo-mo play without quantum stockpiles. There are lots of messages about failed pathing, mostly of my Dwarven Military. Also about double tagging items in food (barrel) stockpiles. I used to have horror of 10 sec freeze and 10 sec slow staggering on 42.04, but converting at least stone and log stockpiles into quantum stockpiles solved it for me. It really did help to lessen the pain of staggering. Nothing else worked, but quantum stockpiles. At least this is what I've thought about it so far. Could this be an issue with military Dwarves dragging their stone haul with them into combat/exercise? Nothing else coming to my mind in this combination of quantum stockpiles, stone+logs, military Dwarves and pathing. These days I use also multiple feeder-stockpiles with 3 wheelbarrows each to avoid military Dwarves slo-mo dragging stones when going on duty to station or to train in barracks. Though dumping stones from inside of wheelbarrows is an irritating activity. Also it helps to evacuate quickly all Dwarves from surface, when siege starts. To stop "a" menu spam, I just need to forbid the wheelbarrows, that were left outside. These days I run all my stockpiles as quantum with exception of food ones. Also I had this 10 sec freeze and 10 sec stagger slo-mo play on a pocket world, but with all other medium settings. Currently I run only all-low settings with my embarks. Maybe this is also a difference? Nothing else comes really to my mind from my experiences playing vanilla DF for last 2 years. Yes, vanilla client. 10-15(10-15) fps here after putting 20k+ blocks in, like a clockwork. Old computer here and can't really afford upgrading it either... Should I upload the errorlog.txt for 0.42.04? There are only 2 types of failure messages there really. Examples here bellow: (1) path fail: dwarf,54,90,169 -> 71,70,11: Id 0000083:Path Goal Seek Station:Station Squad Move at 72,73,11 path fail: dwarf,Construct Building,59,71,171 -> 60,73,171: Id 0000196:Path Goal Main Job Building:Station Mill Building at 58,78,170 path fail: elk bird,67,136,115 -> 68,135,114: Id 0001290:Path Goal Seek Station Flood:Station Owner at 68,135,114 loop path fail: <same square> dwarf,Dump Item,63,59,169 -> 64,60,169: Id 0000677:Path Goal Seek Item for Storage:Station Mill Building at 37,88,171 path fail: troglodyte,66,103,115 -> 56,102,114: Id 0001688:Path Goal Marauder Mill:Station Marauder Mill at 56,102,114 (2) DOUBLE TAGGED ITEM: Store Item in Stockpile, quarry bush leaves [2] DOUBLE TAGGED ITEM: Store Item in Stockpile, pecan wood barrel <0000005> DOUBLE TAGGED ITEM: Store Item in Stockpile, (steel anvil) EDIT: This 0.42.04 errorlog.txt is significantly larger then overage. I have however even earlier version 0.40.24. I didn't use there quantum stockpiles either and I think the significant stagger there was too. All errors in 0.40.24 error.log.txt are double tagged items though or solemnly loop path fails. Mostly double tagged items though. Also I have only 1 save there, the last save of my first ever embark. Later on, I was saving region directory more frequently... for reasons... :)

(0037374) sionlife (reporter) 2017-12-22 08:30	reply to lethosor latest comment: I cleared both my errorlog.txt and gamelog.txt files and ran the stuttering for a few minutes. I rechecked both files and the errorlog.txt was still empty and the gamelog.txt had just 5 lines. So I don't think the problem is being reported in these files.

(0037376) Sarmatian123 (reporter) 2017-12-22 10:30	I will agree with sionlife on this one. Gamelog.txt is not reporting on unseen combat from not discovered yet caverns and I've seen combat "r"-menu growing during stuttering moments only from combat I could see with my own eyes. The big stutter of 10 sec screen freeze + 10 sec of stutter afterward is lessened, if not solved, with quantum stockpiles. I have no clue why. It just works like that in DF. The stone hauling seems to be just an inconvenience. I fail to see how it can cause stutter. Though error flood in errorlog.txt seems somehow slightly correlated with this inconvenience. For some reason.

(0037377) Suspriscus (reporter) 2017-12-22 11:42	Cleared errorlog.txt and gamelog.txt, then ran each save for a few minutes. Save 1: errorlog, 1KB: Two lines about double tagged items, and one about a path fail. gamelog, 16KB: Combat between a forgotten beast and another creature in a cavern. A lot of dwarf vomiting. The combat stopped, stuttering continued. Save 2: errorlog, 0KB: Nothing gamelog, 75KB: Nothing for a few minutes, then a lot of lines of "combat" from dwarfs training. Again, stuttering didn't seem correlated to this. Save 3: errorlog, 0KB: Nothing gamelog, 7KB: Nothing for the first few minutes, then a dwarf starts hunting a wild boar. Again, doesn't seem correlated to the stuttering.

(0037379) Sarmatian123 (reporter) 2017-12-22 16:18	@Suspriscus, Did you discover all 3 caverns? They tend to be at lv80, lv40 and lv30. Did you uncover vision in entirety of all those 3 caverns? On my embarks reports about forgottens' arrival I got only, when they entered tile I had vision on. I never seen any predators in discovered lava sea and hell does not fight hell. Also... Did hell get road to surface/cavern/lava in some other embark in the same world? I never let those roaches out. Inside there is nothing worth digging for anyhow.

(0037380) lethosor (manager) 2017-12-22 16:27	I asked because I saw someone on Reddit report a log that was around a gigabyte in size, and was having similar issues to this. Thanks for checking on that - this issue, at least, doesn't appear to be due to that, then.

(0037381) Suspriscus (reporter) 2017-12-22 18:04	@Sarmatian123 I uncovered all three caverns in my first and second saves, not all revealed. In the third I didn't discover the caverns at all, though.

(0037388) GraphicH (reporter) 2017-12-23 12:42	Another save showing this issue: http://dffd.bay12games.com/file.php?id=13339 [^] I have season backups, the issue began about year 503 and got progressively worse until finally getting into an unplayable state in 504.

(0037406) thvaz (reporter) 2017-12-26 20:30	Another save, for 0.44.03 http://dffd.bay12games.com/file.php?id=13349 [^] My computer should handle this fortress without any problem, but the game is stuttering heavily and sometimes it stops responding (but it recovers after a minute or two)

(0037448) Sarmatian123 (reporter) 2017-12-31 00:01 edited on: 2017-12-31 00:44	For me severe stuttering is solved by using quantum stockpiles. Dunno why or how it works, but it works. I still get freezes of screen during combat. However my fps got halved and almost unnoticeable light stutter appeared, when I allowed 10 hunters hunt freely and to have in my 110 Dwarves fortress 20-30 military Dwarves training constantly (8 squads, training schedule alike "10-2-2-2-2-2-10-2-2-2-2-2"). Now, that I was pointed at those message files in DF directory, I took notice about something strange. Appearance of light stutter is combined with strange limitation of announcements on screen. Announcements from being few pages long got short to barely 6-8 lines of latest messages. All caverns are still hidden on my embark and I discovered only small part of lava sea. There is a bug from 2010. A minor issue. http://www.bay12games.com/dwarves/mantisbt/view.php?id=1430 [^] It seems announcements and reports share common stack of 3000 messages. However when I looked in reports and they almost all were empty too. Like announcements. Where messages of announcements and reports go? Those txt files on harddrive are not growing exponentially either. Why there is light stutter? I understand combat/sparing/hunting could be causing this light stutter, but why then the quantum stockpiles are magically solving the most heavy and severe case of it? How quantum stockpiles are related to sparing&hunting? EDIT: MY BAD! GAMELOG.TXT exploded! It grew suddenly to 22.5mb. HUGE! Like 80% of all of its content now are the sparing messages. It loads into my mousepad text editor like for entire 5 minutes.

(0037449) Sarmatian123 (reporter) 2017-12-31 00:40 edited on: 2017-12-31 01:11	WOW! I renamed gamelog.txt. Created new 0 size gamelog.txt. Started DF anew. My fps magically went up from 7(7)fps to 13(13)fps. just like that. 14(14)-17(17) is what I had in fps when I finished construction of my 20k+ blocks fortress. I checked on announcements... and there were those few last announcements still on. I checked on reports... and there were those few reports still on. However I looked every single of those 4 sparing entries and it seems each was 8-9 pages long and one 18 pages long. The light stutter still on. Dwarves still sparring ;) EDIT: GAMELOG.TXT is being written with speed of 1-3kb/1sec. 10 military Dwarves activity: spar. Also my DF speed went down again to around 8-13 fps. It varies. It is higher then before with flat 7(7) fps. Light stutter as well as sparing still continues. :) 15(15)-16(16) fps! Removing old 22.5mb gamelog.txt was huge to boost fps in my DF. PS. How quantum stockpiles are related to sparing? Is it errorlog.txt thingy? So without quantum stockpiles my errorlog.txt would be few gb huge by now?

(0037461) Orkel (reporter) 2018-01-01 16:16	Here's my save: http://dffd.bay12games.com/file.php?id=13370 [^] At only 73 dwarves, my FPS suddenly dropped to 50 from 140, and there is heavy stutter. Like, it is not smooth at all, the movement of dwarves is jagging heavily. I have done everything I can think of to get it back up again, like killing every single animal and pet on the map (including the caverns using dfhack), making sure no doors are locked, removing all spatters on the map, killing wounded dwarves in their beds, blocking the flow of the brook and underground water using obsidian so they don't flow out of the map, deleting the gamelog.txt, atomsmashing corpses, killing all visitors to the fortress, etc.

(0037463) Sarmatian123 (reporter) 2018-01-01 17:34	Orkel: Did you turn all, except food stuff, stockpiles into quantum stockpiles and then delete errorlog.txt too?

(0037475) Orkel (reporter) 2018-01-02 13:55 edited on: 2018-01-02 13:55	Not quantum stockpiles, but I deconstructed all workshops and stockpiles as a whole (didn't fix it). Even if I made quantum stockpiles it is unlikely to be the cause, since I've run bigger forts with more FPS and much, much bigger stockpiles before this fort.

(0037477) Sarmatian123 (reporter) 2018-01-02 16:03	http://dwarffortresswiki.org/index.php/DF2014:Maximizing_framerate#Fortress_Design [^] "The quantity of items in any particular stack doesn't affect framerate so much as the number of stacks in general, due to the resultant impact on hauling, stockpiles, pathfinding and other CPU-intensive tasks." As you can read above from Wiki's quote, deconstructing stockpiles and workshops are not the same as using quantum stockpiles. I would suggest, you use "d-b-d" to mass mark items for dump with "i"-menu designated 1x1 dump area. Simple even for a beginner to quantum stockpiling. You will need later to allow those items too. Then read on Wiki about quantum stockpiles and build some wooden minecarts and using one or two 11x11 feeder stockpiles create "b-c-S" track stops for automated minecart auto-dump quantum stockpiling. Quantum Stockpiling, generally speeds your frame rate, even if you still experience stutter. I wish there was some designation to mark entrance(s) in a burrow for the pathfinding algorithm. It would save on some cpu. I doubt pathfinding algorithm even uses burrow designations currently. I wish we could assign cats to burrows, like we can with Dwarves. I do not think killing pest outsidefortress makes sense. Pest auto-regenerates. I hope there is some coded limit to the pest levels.

(0037479) sionlife (reporter) 2018-01-02 19:11 edited on: 2018-01-02 19:14	I would rather the cause of the stuttering be found and fixed rather than for everybody to start using QSP and brush the problem under the carpet. In any case, I have seen an example of low fps happen right after the embark in the first 10 seconds. There are no stockpiles and just the items on the wagon. QSP is not the answer we are looking for here.

(0037480) Sarmatian123 (reporter) 2018-01-02 21:14	Once (40.24 or 42.04 it was I think) I have seen my fps stop almost dead after breaching a cave... The stutter I had was nightmarish. I was under impression my computer is crashing. Then miraculously fps came back to life after I atom smashed under a bridge 3 times contents of time after time completely filled 11x11 refuse. Then I had repeat of the same after breaching 2nd cave. My cats were very busy then. I even suggested that maybe vermin has something to do with this stutter/low fps. Now, I do not even breach caves any more and am happy if I can play at 10+ fps. There are plenty of issues causing low fps. You can read on Wiki how to maximize the frame rate. Stutter you always will have with combat, hunting and sparing anyhow. So better to have light stutter, then unplayable stutter. Unplayable stutter is caused by far more issues then just those mentioned by me. It seems DF gameplay begs for optimization and smoothing rough corners on some of those glitches. This is why this bug/issue won't be fixed any time soon. Surely not in this year. However now folks should know to delete those log.txt files in df directory to get at least some fps back. Besides QSP of course. :)

(0037508) RudeGuyGames (reporter) 2018-01-04 16:38	Here's a save with a new embark that has "inherited" the stuttering. I have tried creating a new world, and starting a new embark there got rid of the stuttering for the time being, so it seems to be a world-specific issue. http://dffd.bay12games.com/file.php?id=13387 [^]

(0037510) Toady One (administrator) 2018-01-05 04:21	Found two new problems that fixed the really bad stuttering on the saves I downloaded here; from e.g. five second pauses to smooth play. There are still some minor older issues (for instance, hf births on the day can load files and cause a little tick, and it's not straightforward to change that, but that was in 0.43 and earlier versions).

Issue History
Date Modified	Username	Field	Change
2017-12-17 13:39	Suspriscus	New Issue
2017-12-17 15:13	lethosor	Tag Attached: Save Included
2017-12-17 15:14	lethosor	Note Added: 0037346
2017-12-17 15:14	lethosor	Assigned To	=> lethosor
2017-12-17 15:14	lethosor	Status	new => acknowledged
2017-12-18 14:34	Sarmatian123	Note Added: 0037350
2017-12-18 14:39	Sarmatian123	Note Edited: 0037350	View Revisions
2017-12-19 06:21	sionlife	Note Added: 0037354
2017-12-19 06:22	sionlife	Issue Monitored: sionlife
2017-12-19 06:40	sionlife	Note Edited: 0037354	View Revisions
2017-12-20 08:59	sionlife	Note Added: 0037363
2017-12-21 19:21	lethosor	Relationship added	related to 0009109
2017-12-21 19:24	lethosor	Note Added: 0037370
2017-12-21 19:25	lethosor	Issue Monitored: Sarmatian123
2017-12-21 19:25	lethosor	Note Added: 0037371
2017-12-21 19:25	lethosor	Status	acknowledged => needs feedback
2017-12-21 19:27	lethosor	Relationship added	related to 0010444
2017-12-21 19:31	lethosor	Relationship added	related to 0009554
2017-12-22 04:09	Sarmatian123	Note Added: 0037373
2017-12-22 04:34	Sarmatian123	Note Edited: 0037373	View Revisions
2017-12-22 08:30	sionlife	Note Added: 0037374
2017-12-22 10:30	Sarmatian123	Note Added: 0037376
2017-12-22 11:42	Suspriscus	Note Added: 0037377
2017-12-22 11:42	Suspriscus	Status	needs feedback => assigned
2017-12-22 12:56	Suspriscus	Issue Monitored: Suspriscus
2017-12-22 12:56	Suspriscus	Issue End Monitor: Suspriscus
2017-12-22 16:02	Suspriscus	Issue Monitored: Suspriscus
2017-12-22 16:18	Sarmatian123	Note Added: 0037379
2017-12-22 16:27	lethosor	Note Added: 0037380
2017-12-22 16:27	lethosor	Status	assigned => acknowledged
2017-12-22 18:04	Suspriscus	Note Added: 0037381
2017-12-23 12:42	GraphicH	Note Added: 0037388
2017-12-25 15:03	lethosor	Relationship added	related to 0008076
2017-12-26 20:30	thvaz	Note Added: 0037406
2017-12-31 00:01	Sarmatian123	Note Added: 0037448
2017-12-31 00:04	Sarmatian123	Note Edited: 0037448	View Revisions
2017-12-31 00:04	Sarmatian123	Note Edited: 0037448	View Revisions
2017-12-31 00:14	Sarmatian123	Note Edited: 0037448	View Revisions
2017-12-31 00:25	Sarmatian123	Note Edited: 0037448	View Revisions
2017-12-31 00:26	Sarmatian123	Note Edited: 0037448	View Revisions
2017-12-31 00:40	Sarmatian123	Note Added: 0037449
2017-12-31 00:41	Sarmatian123	Note Edited: 0037449	View Revisions
2017-12-31 00:42	Sarmatian123	Note Edited: 0037449	View Revisions
2017-12-31 00:44	Sarmatian123	Note Edited: 0037448	View Revisions
2017-12-31 00:52	Sarmatian123	Note Edited: 0037449	View Revisions
2017-12-31 00:53	Sarmatian123	Note Edited: 0037449	View Revisions
2017-12-31 00:54	Sarmatian123	Note Edited: 0037449	View Revisions
2017-12-31 00:55	Sarmatian123	Note Edited: 0037449	View Revisions
2017-12-31 00:56	Sarmatian123	Note Edited: 0037449	View Revisions
2017-12-31 00:59	Sarmatian123	Note Edited: 0037449	View Revisions
2017-12-31 01:11	Sarmatian123	Note Edited: 0037449	View Revisions
2018-01-01 16:16	Orkel	Note Added: 0037461
2018-01-01 17:34	Sarmatian123	Note Added: 0037463
2018-01-02 13:55	Orkel	Note Added: 0037475
2018-01-02 13:55	Orkel	Note Edited: 0037475	View Revisions
2018-01-02 16:03	Sarmatian123	Note Added: 0037477
2018-01-02 19:11	sionlife	Note Added: 0037479
2018-01-02 19:14	sionlife	Note Edited: 0037479	View Revisions
2018-01-02 21:14	Sarmatian123	Note Added: 0037480
2018-01-04 16:38	RudeGuyGames	Note Added: 0037508
2018-01-05 04:21	Toady One	Note Added: 0037510
2018-01-05 04:21	Toady One	Status	acknowledged => resolved
2018-01-05 04:21	Toady One	Fixed in Version	=> Next Version
2018-01-05 04:21	Toady One	Resolution	open => fixed
2018-01-05 04:21	Toady One	Assigned To	lethosor => Toady One
2019-01-04 23:05	Sarmatian123	Issue End Monitor: Sarmatian123