That's a complicated issue.
Catacomb is an MS-DOS game, running with the help of DOSBox emulator. Since the original resolution of such games is often poor by today standards, it supports two ways to increase display size:
- "scalers" that transform the original pixels into 2x2 or 3x3 pixel matrices using various algorithms, from very simple ones to quite complex ones, to produce pleasing displays, already 2 or 3 times larger than the original
- hardware scaling, that take the image generated above to scale it to the final resolution, usually the screen resolution.
Among the scalers, I've found the hq2x one to often give the results I liked the most, even if it can depend on the game, and I used in most of my DOSBox based install scripts. It uses neighbour pixels to round the edges, using lookups in manually optimized patterns. Anyway.
A problem I've since discovered, is that not all scalers are compatible with the next hardware scaling step, and the hq*x scalers are among the incompatible ones. What that means is that when you use them, the display will be locked to the 2x or 3x resolution provided by the scaler alone.
What you can try, for the lack of a GUI to tweak DOSBox settings at the moment, is to open ~/.PlayOnLinux/wineprefix/Catacombs_gog/playonlinux_dos.cfg file with your favorite text editor, and modify the
render_scaler=hq2x
line to, say,
render_scaler=normal2x
which supports hardware scaling. If it works you can try the other scalers, but from what I remember only the simplest ones support hardware scaling, which is a shame... :(