An open letter to Peter Cohen
(e.g., MAC address, hostname, email address). Lax System and Network Administration— The SANS Institute identifies the most frequently-targeted OS, application and network components in its annual Top 20 list—but you could easily compile a list of common configuration errors that is twice as long and causes more security inci-dents. Tor is an encrypted anonymising network that makes it harder to intercept internet communications, or see where communications are coming from or going to. In order to use the WikiLeaks public submission system as detailed above you can download the Tor Browser Bundle, which is a Firefox-like browser available for Windows, Mac OS X and GNU/Linux and pre-configured to connect using the.
Dear Peter,
As you know, I once penned Macworld'sGame Room column. And, now blessed with that beat, you may be starting to feel the way I did once I passed the column along to Andy Ihnatko—if you ever see another computer game, it will be too soon. The once unrelenting joy of vaporizing one cootie after another became a terrible grind.
Having had nearly a decade-long break from the column, my interest in gaming has started to return. Sure, it's only been resurrected to the point where I've purchased the reissue of Return to Dark Castle and LittleWing's Monster Fair (so sue me, I've got a nostalgic streak and I'm still a sucker for pinball), but at least there's some spark.
What rekindled my once-dead romance with gaming may surprise you. It was my iPhone. I've jailbroken the thing since just about Day 1 and when I have a spare couple of minutes, it's not unusual to find me banging away on Mahjong 2.
I mention this game specifically because I noticed that the iTunes Store now offers updated versions of iPod games that were pulled from the Store's shelves because they weren't compatible with the 3G iPod nano and iPod classic. Among those games is Electronic Arts' Mahjong.
Given that I've had recent experience with the iPhone's version of the game, I thought I'd give the Apple-sanctioned Mahjong a spin (quite literally, thanks to the iPod's clickwheel). Having done so I was prompted to write to you because you, more than anyone I know, might have an informed answer to this question:
iPod gaming kind of sucks, right?
And I suggest that for these reasons:
- An iPod's screen—even a 5G or iPod classic—is awfully small for someone who doesn't have 14-year-old eyes.
- The clickwheel is great for quickly moving through lists but as a game controller, ick.
- The eye candy on some of these things is nice enough, but try playing these games without sound (as you do with games on jailbroken phones). Maybe it's just me, but without the accompanying noise some of these games don't stand up very well.
I admit it, I was among those crying out for more iPod games, but having seen what's possible on the iPhone and iPod touch (also jailbroken)—larger screen, acceleration, touch-interface—it's hard for me to return to a 'traditional' iPod and gain much pleasure from spinning a wheel and clicking a button. And I have to think that it will be just as hard for those who own both an iPhone/iPod touch and a clickwheel iPod when third-party games come to the iPhone and touch this summer.
Will the iPhone 2.0 software and the games that will pour in with it spell the end for gaming on traditional iPods? You now bear the master's mantle, please set me straight.
Your friend and colleague,
Chris
Read Peter Cohen's response.
Until now, the Platform::Sdl2Application was the go-to solution for mostplatforms including the web and mobile. However, not everybody needs all thefeatures SDL provides and, especially on Emscripten, apart from simplifyingporting it doesn't really add anything extra on top. On the contrary, theadditional layer of translation between HTML5 and SDL APIs increases theexecutable size and makes some features unnecessarily hard to access.
To solve that, the new Platform::EmscriptenApplication, contributed inmosra/magnum#300 by @Squareys, is using Emscripten HTML5 APIsdirectly, opening new possibilities while making the code smaller and moreefficient.
'SDL2' vs SDL2
Since there's some confusion about SDL among Emscripten users, let's clarifythat first. Using SDL in Emscripten is actually possible in two ways — theimplicit support, implemented in library_sdl.js,gives you a slightly strange hybrid of SDL1 and SDL2 in a relatively smallpackage. Not all SDL2 APIs are present there, on the other hand it has enoughfrom SDL2 to make it a viable alternative to the SDL2 everyone is used to. Thisis what Platform::Sdl2Application is using.
The other way is a 'full SDL2', available if you pass -s USE_SDL=2
to thelinker. Two years ago we tried to remove all Emscripten-specific workaroundsfrom Platform::Sdl2Application by switching to this full SDL2, butquickly realized it was a bad decision — in total it removed 30 lines ofcode, but caused the resulting code to be almost 600 kB larger. The sizeincrease was so serious that it didn't warrant the very minor improvements incode maintainability. For the record, the original pull request is archived atmosra/magnum#218.
The SDL-free EmscriptenApplication
All application implementations in Magnum strive for almost full APIcompatibility, with the goal of making it possible to use an implementationoptimal for chosen platform and use case. This was already the case withPlatform::GlfwApplication and Platform::Sdl2Application, whereswitching from one to the other is in 90% cases just a matter of using adifferent #include
and passing a different component to CMake'sfind_package()
.
The new Platform::EmscriptenApplication continues in this fashion and weported all existing examples and tools that formerly usedPlatform::Sdl2Application to it to ensure it works in broad use cases.Apart from that, the new implementation fixes some of the long-standing issueslike miscalculated event coordinates on mobile web browsers or theDelete key leaking through text input events.
Only two widely used APIs are missing from it right now — thePlatform::Sdl2Application::tickEvent() andPlatform::Sdl2Application::setSwapInterval(). The former will getadded together with an equivalent in GLFW application, while the secondwill be exposed differently, allowing to use the extended browser APIs.Right now it's enough to#ifdef
around it, as browsers, unlikemost desktop platforms, enable VSync by default.Power-efficient idle behavior
Since the very beginning, all Magnum application implementations default toredrawing only when needed in order to save power — because Magnum is notjust for games that have to animate something every frame, it doesn't makesense to use up all system resources by default. While this is simple toimplement efficiently on desktop apps where the application has the fullcontrol over the main loop (and thus can block indefinitely waiting for aninput event), it's harder in the callback-based browser environment.
The original Platform::Sdl2Application makes use ofemscripten_set_main_loop(),which periodically calls window.requestAnimationFrame()in order to maintain a steady frame rate. For apps that need to redraw onlywhen needed this means the callback will be called 60 times per second only tobe a no-op. While that's still significantly more efficient than drawingeverything each time, it still means the browser has to wake up 60 times persecond to do nothing.
Whelm Mac Os Download
Platform::EmscriptenApplication instead makes use ofrequestAnimationFrame()directly — the next animation frame is implicitly scheduled, but cancelledagain after the draw event if the app doesn't wish to redraw immediately again.That takes the best of both worlds — redraws are still VSync'd, but thebrowser is not looping needlessly if the app just wants to wait with a redrawfor the next input event. To give you some numbers, below is a ten-secondoutput of Chrome's performance monitor comparing SDL and Emscripten appimplementation waiting for an input event. You can reproduce this with theMagnum Player — no matter how complexanimated scene you throw at it, if you pause the animation it will use as muchCPU as a plain static text web page.
DPI awareness revisited
Arguably to simplify porting, the Emscripten SDL emulation recalculates allinput event coordinates to match framebuffer pixels. The actual DPI scaling(or device pixel ratio) is then being exposed through dpiScaling(),making it behave the same as Linux, Windows and Android on high-DPI screens. Incontrast, HTML5 APIs behave like macOS / iOS andPlatform::EmscriptenApplication follows that behavior —framebufferSize()thus matches device pixels while windowSize()(to which all events are related) is smaller on HiDPI systems. For moreinformation, check out the DPI awareness docs.
It's important to note that even though different platforms expose DPIawareness in a different way, Magnum APIs are designed in a way that makesit possible to have the same code behave correctly everywhere. Theseparation into dpiScaling(),framebufferSize() andwindowSize() propertiesis mainly for a more fine-grained control where needed.Executable size savings
Because we didn't end up using the heavyweight 'full SDL2' in the first place,the difference in executable size is nothing extreme — in total, in a ReleaseWebAssembly build, the JS size got smaller by about 20 kB, while the WASM filestays roughly the same.
Minimal runtime, or brain surgery with a chainsaw
On the other hand, since the new application doesn't use any of the emscripten_set_main_loop()
APIs from library_browser.js
, it makes ita good candidate for playing with the relatively recentMINIMAL_RUNTIME feature of Emscripten.Now, while Magnum is moving in the right direction, it's not yet in a statewhere this would 'just work'. Supporting MINIMAL_RUNTIME
requires eithermoving fast and breaking lots of things or have the APIs slowly evolve into astate that makes it possible. Because reliable backwards compatibility andpainless upgrade path is a valuable asset in our portfolio, we chose thelatter — it will eventually happen, but not right now. Another reason is thatwhile Magnum itself can be highly optimized to be compatible with minimalruntime, the usual application code is not able to satisfy those requirementswithout removing and rewriting most third-party dependencies.
That being said, why not spend one afternoon with a chainsaw and trydemolishing the code to see what could come out? It's however important tonote that MINIMAL_RUNTIME
is still a very fresh feature and thus it's verylikely that a lot of code will simply not work with it. All the discoveredproblems are listed below because at this point there are no results at allwhen googling them, so hopefully this helps other people stuck in similarplaces:
Mac Os Download
- std::getenv() or the
environ
variable (used byUtility::Arguments) results inwriteAsciiToMemory()
beingcalled, which is right now explicitly disabledfor minimal runtime (and thus you either get a failure at runtime or theClosure Compiler complaining about these names being undefined). SinceEmscripten's environment is just a bunch of hardcoded values and Magnum isusing Node.js APIs to get the real values for command-line apps anyway,solution is to simply not use those functions. - Right now, Magnum is using C++ iostreams on three isolated places(Utility::Debug being the most prominent) and those uses aregradually being phased off. On Emscripten, using anything that evenremotely touches them will make the backend emit calls to
llvm_stacksave()
andllvm_stackrestore()
. The JavaScript implementationsthen callstackSave()
andstackRestore()
which however do notget pulled in inMINIMAL_RUNTIME
, again resulting in either a runtimeerror every time you call into JS (so also allemscripten_set_mousedown_callback()
functions) or when you use theClosure Compiler. After wasting a few hours trying to convince Emscriptento emit these two by adding_llvm_stacksave__deps:['$stackSave']
theultimate solution was to kill everything stream-related. Consideringeveryone who's interested inMINIMAL_RUNTIME
probably did that already,it explains why this is another ungoogleable error. - If you use C++ streams, the generated JS driver file contains a fullJavaScript implementation of
strftime()
and the only way to get ridof it is removing all stream usage as well. Grep your JS file forMonday
— if it's there, you have a problem. - JavaScript Emscripten APIs like
dynCall()
orallocate()
are notavailable and putting them into eitherEXTRA_EXPORTED_RUNTIME_METHODS
orRUNTIME_FUNCS_TO_IMPORT
either didn't do anything or moved theerror into a different place. For the former it was possible to work aroundit by directly calling one of its specializations (in that particular casedynCall_ii()
), the second resulted in a frustrated tableflip and therelevant piece of code getting cut off.
Dodge the creeps redux mac os. Below is a breakdown of various optimizations on a minimal application thatdoes just a framebuffer clear, each step chopping another bit off the totaldownload size. All sizes are uncompressed, built in Release mode with -Oz
,--llvm-lto 1
and --closure 1
. Later on in the process,Bloaty McBloatFace experimentalWebAssembly support was used to discover what functions contribute the most to finalcode size.
That being said, why not spend one afternoon with a chainsaw and trydemolishing the code to see what could come out? It's however important tonote that MINIMAL_RUNTIME
is still a very fresh feature and thus it's verylikely that a lot of code will simply not work with it. All the discoveredproblems are listed below because at this point there are no results at allwhen googling them, so hopefully this helps other people stuck in similarplaces:
Mac Os Download
- std::getenv() or the
environ
variable (used byUtility::Arguments) results inwriteAsciiToMemory()
beingcalled, which is right now explicitly disabledfor minimal runtime (and thus you either get a failure at runtime or theClosure Compiler complaining about these names being undefined). SinceEmscripten's environment is just a bunch of hardcoded values and Magnum isusing Node.js APIs to get the real values for command-line apps anyway,solution is to simply not use those functions. - Right now, Magnum is using C++ iostreams on three isolated places(Utility::Debug being the most prominent) and those uses aregradually being phased off. On Emscripten, using anything that evenremotely touches them will make the backend emit calls to
llvm_stacksave()
andllvm_stackrestore()
. The JavaScript implementationsthen callstackSave()
andstackRestore()
which however do notget pulled in inMINIMAL_RUNTIME
, again resulting in either a runtimeerror every time you call into JS (so also allemscripten_set_mousedown_callback()
functions) or when you use theClosure Compiler. After wasting a few hours trying to convince Emscriptento emit these two by adding_llvm_stacksave__deps:['$stackSave']
theultimate solution was to kill everything stream-related. Consideringeveryone who's interested inMINIMAL_RUNTIME
probably did that already,it explains why this is another ungoogleable error. - If you use C++ streams, the generated JS driver file contains a fullJavaScript implementation of
strftime()
and the only way to get ridof it is removing all stream usage as well. Grep your JS file forMonday
— if it's there, you have a problem. - JavaScript Emscripten APIs like
dynCall()
orallocate()
are notavailable and putting them into eitherEXTRA_EXPORTED_RUNTIME_METHODS
orRUNTIME_FUNCS_TO_IMPORT
either didn't do anything or moved theerror into a different place. For the former it was possible to work aroundit by directly calling one of its specializations (in that particular casedynCall_ii()
), the second resulted in a frustrated tableflip and therelevant piece of code getting cut off.
Dodge the creeps redux mac os. Below is a breakdown of various optimizations on a minimal application thatdoes just a framebuffer clear, each step chopping another bit off the totaldownload size. All sizes are uncompressed, built in Release mode with -Oz
,--llvm-lto 1
and --closure 1
. Later on in the process,Bloaty McBloatFace experimentalWebAssembly support was used to discover what functions contribute the most to finalcode size.
Operation | JS size | WASM size |
---|---|---|
Initial state | 52.1 kB | 226.3 kB |
Enabling minimal runtime 1 | 36.3 kB | 224.5 kB |
Additional slimming flags 2 | 35.7 kB | 224.5 kB |
Disabling filesystem 3 | 19.4 kB | 224.5 kB |
Chopping off all C++ stream usage | 14.7 kB | 83.6 kB |
Enabling CORRADE_NO_ASSERT | 14.7 kB | 75.4 kB |
Removing a single use of std::sort()4 | 14.7 kB | 69.3 kB |
Removing one std::unordered_map4 | 14.7 kB | 62.6 kB |
Using emmalloc instead of dlmalloc 5 | 14.7 kB | 56.3 kB |
Removing all printf() usage 6 | 14.7 kB | 44 kB (estimate) |
Mac Os Versions
- 1.
- ^
-s MINIMAL_RUNTIME=2 -s ENVIRONMENT=web -lGL
plus temporarilyenabling also-s IGNORE_CLOSURE_COMPILER_ERRORS=1
in order to makeClosure Compiler survive undefined variable errors due to iostreams andother, mentioned above - 2.
- ^
-s SUPPORT_ERRNO=0 -s GL_EMULATE_GLES_VERSION_STRING_FORMAT=0 -s GL_EXTENSIONS_IN_PREFIXED_FORMAT=0 -s GL_SUPPORT_AUTOMATIC_ENABLE_EXTENSIONS=0 -s GL_TRACK_ERRORS=0 -s DISABLE_DEPRECATED_FIND_EVENT_TARGET_BEHAVIOR=1
— basically disabling what's enabled by default. In particular, theGL_EXTENSIONS_IN_PREFIXED_FORMAT=0
is not supported by Magnum rightnow, causing it to not report any extensions, but that can be easily fixed.The result of disabling all these is … underwhelming. - 3.
- ^
-s FILESYSTEM=0
, makes Emscripten not emit any filesystem-relatedcode. Magnum provides filesystem access through various APIs(Utility::Directory, GL::Shader::addFile(),Trade::AbstractImporter::openFile(), …) and at the moment there'sno possibility to compile all these out, so this is a nuclear option thatworks. - 4.
- ^ abGL::Context uses a std::sort() and astd::unordered_map to check for extension presence and print theirlist in the engine startup log. It was frightening to see a removal of asingle std::sort() causing a 10% drop in executable size — sinceWebGL has roughly two dozens extensions (compared to > 200 on desktop andES), maybe a space-efficient alternative implementation could be donefor this target instead.
- 5.
- ^Doug Lea‘smalloc() is a general-purpose allocator, used byglibc among others. It's very performant and a good choice for code thatdoes many small allocations (std::unordered_map, I'm looking atyou). The downside is its larger size, and code doing fewer largerallocations might want to use
-s MALLOC=emmalloc
instead. We don'tpretend Magnum is at that state yet, but other projectssucessfully switched to it, shavingmore bytes off the download size. - 6.
- ^ After removing all of the above, std::printf() internals startedappearing at the top of Bloaty's size report, totalling at about 10% of theexecutable size. Magnum doesn't use it anywhere directly and all transitiveusage of it was killed together with iostreams; further digging revealedthat it gets called from libc++'s abort_message(),for example when aborting due to a pure virtual function call. Independentmeasurement showed that std::printf() is around 12 kB of additionalcode compared to std::puts(), mainly due to the inherent complexityof floating-point string conversion. It's planned to use the muchsimpler and smaller Ryū algorithmfor Magnum's std::printf() replacement, additionally ensuring thatfloat-to-string conversions can be DCE-dwhen not used. We might be looking into patching Emscripten's libc++ to notuse the expensive implementation in its abort messages.
While all of the above size reductions were done in a hack-and-slash manner,the final executable still initializes and executes properly, clearing theframebuffer and reacting to input events. For reference, check out diffs ofthe chainsaw-surgery
branches in corradeand magnum.
The above is definitely not all that can be done — especially consideringthat removing two uses of semi-heavy STL APIs led to almost 20% save in codesize, there are most probably more of such low hanging fruits. The above taskswere added to mosra/magnum#293 (if not there already) and will getgradually integrated into master
.
Conclusion
Bright times ahead! The new Platform::EmscriptenApplication is the firststep to truly minimal WebAssembly builds and the above hints that it's possibleto have download sizes not too far from code carefully written in plain C.To give a fair comparison, the basic framebuffer clear sample from@floooh‘s Sokol Samples is 42kB in total, while the above equivalent is roughly 59 kB. Using C++(11), butnot overusing it — and that's just the beginning.
Questions? Complaints? Share your opinion on social networks:Twitter,Reddit r/cpp,r/WebAssembly,Hacker News