Dwarf Fortress Bug Tracker - Dwarf Fortress
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0006558Dwarf FortressTechnical -- Input/Keybinding/Macrospublic2014-07-07 21:222014-08-01 12:16
Reporterlue 
Assigned ToToady One 
PrioritynormalSeverityminorReproducibilityalways
StatusresolvedResolutionfixed 
PlatformLinuxOSGentoo LinuxOS Version
Product Version0.40.01 
Target VersionFixed in Version0.40.06 
Summary0006558: "Broken Unicode" warnings for some keybindings
DescriptionHere is the output I receive upon startup of dwarf fortress, up until the title screen appears (with introductory movie playing):

(the LD_PRELOAD is to workaround another, already-reported bug involving libz and libpng)

$ LD_PRELOAD=/lib32/libz.so.1 ./df
ERROR: ld.so: object '/lib32/libz.so.1' from LD_PRELOAD cannot be preloaded: ignored.
ERROR: ld.so: object '/lib32/libz.so.1' from LD_PRELOAD cannot be preloaded: ignored.
Sound devices available:
OpenAL Soft
Picking OpenAL Soft. If your desired device was missing, make sure you have the appropriate 32-bit libraries installed. If you wanted a different device, configure ~/.openalrc appropriately.
Perfect OpenAL context attributes GET
Loading bindings from data/init/interface.txt
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
New window size: 640x300
Font size: 8x12
Resizing grid to 80x25
Resizing font to 8x12

Resetting textures
Resizing font to 8x12

This appears to be related to data/init/interface.txt
Steps To ReproduceStart up dwarf fortress. This shows up instantaneously.
Additional InformationA comparison for reference. This is a relevant snippet of the 0.34.11 interface.txt, as seen by the program `less`:

[BIND:STRING_A127:REPEAT_SLOW]
[KEY:^?]
[BIND:STRING_A128:REPEAT_SLOW]
[KEY:<U+0080>]
[BIND:STRING_A129:REPEAT_SLOW]
[KEY:<U+0081>]
[BIND:STRING_A130:REPEAT_SLOW]
[KEY:<U+0082>]
[BIND:STRING_A131:REPEAT_SLOW]
[KEY:<U+0083>]
[BIND:STRING_A132:REPEAT_SLOW]
[KEY:<U+0084>]
[BIND:STRING_A133:REPEAT_SLOW]
[KEY:<U+0085>]
[BIND:STRING_A134:REPEAT_SLOW]
[KEY:<U+0086>]
[BIND:STRING_A135:REPEAT_SLOW]
[KEY:<U+0087>]
[BIND:STRING_A136:REPEAT_SLOW]

Here is the same snippet in 0.40.01:

[BIND:STRING_A127:REPEAT_SLOW]
[KEY:^?]
[BIND:STRING_A128:REPEAT_SLOW]
[KEY:?]
[BIND:STRING_A129:REPEAT_SLOW]
[KEY:<81>]
[BIND:STRING_A130:REPEAT_SLOW]
[KEY:?]
[BIND:STRING_A131:REPEAT_SLOW]
[KEY:?]
[BIND:STRING_A132:REPEAT_SLOW]
[KEY:?]
[BIND:STRING_A133:REPEAT_SLOW]
[KEY:?]
[BIND:STRING_A134:REPEAT_SLOW]
[KEY:?]
[BIND:STRING_A135:REPEAT_SLOW]
[KEY:?]
[BIND:STRING_A136:REPEAT_SLOW]
[KEY:?]

As you can see, some of these special characters have transformed into plain question marks (which won't be reported by the console), while others (only one in this snippet, though there are more) are straight byte values, instead of properly encoded codepoints.
TagsNo tags attached.
Relationships
related to 0003497resolved Toady One incorrect string entry keybindings 
Attached Files
Issue History
Date ModifiedUsernameFieldChange
2014-07-07 21:22lueNew Issue
2014-07-08 21:10Khym ChanurNote Added: 0025238
2014-07-08 21:10Khym ChanurIssue Monitored: Khym Chanur
2014-07-22 14:48QuietustNote Added: 0027248
2014-07-22 16:32lueNote Added: 0027255
2014-07-22 19:36FootkerchiefNote Added: 0027264
2014-07-22 20:11lethosorNote Added: 0027267
2014-07-22 20:11lethosorAssigned To => lethosor
2014-07-22 20:11lethosorStatusnew => acknowledged
2014-07-22 20:11lethosorNote Edited: 0027267bug_revision_view_page.php?bugnote_id=0027267#r10223
2014-07-22 20:12lethosorNote Edited: 0027267bug_revision_view_page.php?bugnote_id=0027267#r10224
2014-07-22 20:41lueNote Added: 0027273
2014-07-25 05:35Hello71Issue Monitored: Hello71
2014-07-28 16:32lethosorStatusacknowledged => confirmed
2014-07-30 09:24tesoulNote Added: 0027956
2014-07-30 12:24lueNote Added: 0027974
2014-08-01 12:16Toady OneStatusconfirmed => resolved
2014-08-01 12:16Toady OneFixed in Version => Next Version
2014-08-01 12:16Toady OneResolutionopen => fixed
2014-08-01 12:16Toady OneAssigned Tolethosor => Toady One
2015-07-28 19:52lethosorRelationship addedrelated to 0003497

Notes
(0025238)
Khym Chanur   
2014-07-08 21:10   
A list of characters which actually shows something:

Broken unicode: [KEY:<81>]
Broken unicode: [KEY:<8d>]
Broken unicode: [KEY:<8f>]
Broken unicode: [KEY:<90>]
Broken unicode: [KEY:<9d>]
Broken unicode: [KEY: ]
Broken unicode: [KEY:¡]
Broken unicode: [KEY:¢]
Broken unicode: [KEY:£]
Broken unicode: [KEY:¤]
Broken unicode: [KEY:¥]
Broken unicode: [KEY:¦]
Broken unicode: [KEY:§]
Broken unicode: [KEY:¨]
Broken unicode: [KEY:©]
Broken unicode: [KEY:ª]
Broken unicode: [KEY:«]
Broken unicode: [KEY:¬]
Broken unicode: [KEY:­]
Broken unicode: [KEY:®]
Broken unicode: [KEY:¯]
Broken unicode: [KEY:°]
Broken unicode: [KEY:±]
Broken unicode: [KEY:²]
Broken unicode: [KEY:³]
Broken unicode: [KEY:´]
Broken unicode: [KEY:µ]
Broken unicode: [KEY:¶]
Broken unicode: [KEY:·]
Broken unicode: [KEY:¸]
Broken unicode: [KEY:¹]
Broken unicode: [KEY:º]
Broken unicode: [KEY:»]
Broken unicode: [KEY:¼]
Broken unicode: [KEY:½]
Broken unicode: [KEY:¾]
Broken unicode: [KEY:¿]
Broken unicode: [KEY:À]
Broken unicode: [KEY:Á]
Broken unicode: [KEY:Â]
Broken unicode: [KEY:Ã]
Broken unicode: [KEY:Ä]
Broken unicode: [KEY:Å]
Broken unicode: [KEY:Æ]
Broken unicode: [KEY:Ç]
Broken unicode: [KEY:È]
Broken unicode: [KEY:É]
Broken unicode: [KEY:Ê]
Broken unicode: [KEY:Ë]
Broken unicode: [KEY:Ì]
Broken unicode: [KEY:Í]
Broken unicode: [KEY:Î]
Broken unicode: [KEY:Ï]
Broken unicode: [KEY:Ð]
Broken unicode: [KEY:Ñ]
Broken unicode: [KEY:Ò]
Broken unicode: [KEY:Ó]
Broken unicode: [KEY:Ô]
Broken unicode: [KEY:Õ]
Broken unicode: [KEY:Ö]
Broken unicode: [KEY:×]
Broken unicode: [KEY:Ø]
Broken unicode: [KEY:Ù]
Broken unicode: [KEY:Ú]
Broken unicode: [KEY:Û]
Broken unicode: [KEY:Ü]
Broken unicode: [KEY:Ý]
Broken unicode: [KEY:Þ]
Broken unicode: [KEY:ß]
Broken unicode: [KEY:à]
Broken unicode: [KEY:á]
Broken unicode: [KEY:â]
Broken unicode: [KEY:ã]
Broken unicode: [KEY:ä]
Broken unicode: [KEY:å]
Broken unicode: [KEY:æ]
Broken unicode: [KEY:ç]
Broken unicode: [KEY:è]
Broken unicode: [KEY:é]
Broken unicode: [KEY:ê]
Broken unicode: [KEY:ë]
Broken unicode: [KEY:ì]
Broken unicode: [KEY:í]
Broken unicode: [KEY:î]
Broken unicode: [KEY:ï]
Broken unicode: [KEY:ð]
Broken unicode: [KEY:ñ]
Broken unicode: [KEY:ò]
Broken unicode: [KEY:ó]
Broken unicode: [KEY:ô]
Broken unicode: [KEY:õ]
Broken unicode: [KEY:ö]
Broken unicode: [KEY:÷]
Broken unicode: [KEY:ø]
Broken unicode: [KEY:ù]
Broken unicode: [KEY:ú]
Broken unicode: [KEY:û]
Broken unicode: [KEY:ü]
Broken unicode: [KEY:ý]
Broken unicode: [KEY:þ]
Broken unicode: [KEY:ÿ]
(0027248)
Quietust   
2014-07-22 14:48   
These errors are technically also happening on Windows, though the messages don't seem to show up unless DFHack is installed (in which case they get dumped to stdout.log).
(0027255)
lue   
2014-07-22 16:32   
I finally got around to a small script that fixed the file, the results of which are pasted here: https://gist.github.com/lue/8fa0425f6bb806b18baf [^] . This is assuming, of course, that the keybindings are supposed to correlate with the first 256 codepoints of Unicode (instead of, say, CP 437), which appears to be the case.

Interestingly, in addition to the broken unicode errors and the erroneous question marks, codepoint 123 was a } when it should've been a { . I wonder if anyone ran into confusing problems with that in-game :) .
(0027264)
Footkerchief   
2014-07-22 19:36   
I'm confused, does this problem only happen when using DFHack, or does that just make the error messages visible?
(0027267)
lethosor   
2014-07-22 20:11   
(edited on: 2014-07-22 20:12)
Happens with a clean installation on OS X and Linux (and Windows, according to comments above, although the messages only show up if DFHack is installed).
(0027273)
lue   
2014-07-22 20:41   
For me, the "Broken Unicode" messages are printed to stdout (_not_ stderr). I don't use any utilities.

You can check out the interface.txt file in your installation of DF, and see that the STRING_A keybindings starting with STRING_A128 are either question marks (likely put there by some text editor handling unrecognized input) or raw byte values, as opposed to utf8-encoded codepoints. A123 and A124 were also set incorrectly, for whatever reason.
(0027956)
tesoul   
2014-07-30 09:24   
In linux, open the file interface.txt with gedit as ISO-8859-1 and save as UTF-8
(0027974)
lue   
2014-07-30 12:24   
tesoul: please note that resaving like that won't fix all the keybindings, just the ones where plain byte values managed to survive to the final file. The ones that were substituted with question marks, as well as the { character keybinding, would still be broken.

Also, the file I posted is still valid for 0.40.05 (that is, running `diff` on the result of fixing 0.40.04's file and the result of fixing 0.40.05's led to no output)