Emscripten: Unable to compile a simple Swift file (maybe not possible?)

22

Hi team, playing around with Emscripten in earnest for the first time, not my usual area of work so apologies if this is clearly never going to work. What I'd like to do is compile a simple Swift program to JS:

println("Hello, world!")

So far what I've tried is:

xcrun swift hello_world.swift -emit-bc -o hello_world.bc
emcc hello_world.bc

but that fails because it can't find certain symbols (looks like the standard library isn't present):

Value:   %1 = call { i8*, i64, i64 } @_TFSS37_convertFromBuiltinUTF16StringLiteralfMSSFTBp17numberOfCodeUnitsBw_SS(i8* bitcast ([14 x i16]* @0 to i8*), i64 13)
LLVM ERROR: Unrecognized struct value
Traceback (most recent call last):
  File "/Users/glen/Downloads/emsdk_portable/emscripten/1.16.0/emcc", line 1540, in <module>
    shared.Building.llvm_opt(final, link_opts)
  File "/Users/glen/Downloads/emsdk_portable/emscripten/1.16.0/tools/shared.py", line 1267, in llvm_opt
    assert os.path.exists(target), 'Failed to run llvm optimizations: ' + output
AssertionError: Failed to run llvm optimizations:

Drilling a bit further, and looking at how xcrun swift -v hello_world.swift actually works, I'm able to use this linking command to compile the bitcode to an executable using normal ld.

cp /Applications/Xcode6-Beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/macosx/libswift_stdlib_core.dylib .
/usr/bin/ld hello_world.bc \
              -lSystem -arch x86_64 \
              -L . -rpath . \
              -o hello_world
./hello_world
=> Hello, world!

So, it seems like libswift_stdlib_core.dylib is the only dependency for hello_world.bc to be properly linked to an executable.

But I'm stuck - is there some equivalent between the -L and -rpath flags on ld that I should be passing to emcc? Or is a dylib like that not possible to be used in emscripten? The final command I tried was:

> emcc libswift_stdlib_core.dylib hello_world.bc                                 ~/src/experiments/swift.js • 2.1.0p0
WARNING  root: emcc: cannot find library "swift_stdlib_core"
Value:   %1 = call { i8*, i64, i64 } @_TFSS37_convertFromBuiltinUTF16StringLiteralfMSSFTBp17numberOfCodeUnitsBw_SS(i8* bitcast ([14 x i16]* @0 to i8*), i64 13)
LLVM ERROR: Unrecognized struct value
Traceback (most recent call last):
  File "/Users/glen/Downloads/emsdk_portable/emscripten/1.16.0/emcc", line 1540, in <module>
    shared.Building.llvm_opt(final, link_opts)
  File "/Users/glen/Downloads/emsdk_portable/emscripten/1.16.0/tools/shared.py", line 1267, in llvm_opt
    assert os.path.exists(target), 'Failed to run llvm optimizations: ' + output
AssertionError: Failed to run llvm optimizations:

Hopefully something's possible from here - Swift is a neat high-level language with a nice type system and no garbage collector, which makes me think it's a good fit for compiling to JS.

geelen picture geelen  ·  14 Jun 2014

Most helpful comment

28

LLVM 7.0 was released yesterday with a lot of progress for WebAssembly target support. While the backend is still marked as experimental, not as much patching of upstream LLVM repositories should be needed. I've pulled 7.0 updates into Apple swift's fork repositories and rebased the changes in the main swift repository by @patcheng, while also fixing a few compilation issues.

Now the whole fresh toolchain with patches from above comments compiles for me, although I haven't checked yet if the actual WebAssembly support has improved. Going to look into that soon. I'm doing the work in these forks and branches:

https://github.com/maxdesiatov/swift
https://github.com/maxdesiatov/swift-compiler-rt
https://github.com/maxdesiatov/swift-llvm
https://github.com/maxdesiatov/swift-clang
https://github.com/maxdesiatov/lld

MaxDesiatov picture MaxDesiatov  ·  20 Sep 2018

All comments

0

Unfortunately it is not possible to link .dylib files to Emscripten. The reason for that is that the .dylibs already contain native machine code for x86/x64, and Emscripten cannot "go back" and get that to LLVM IR form again. What one would have to do is implement the standard library for Swift and compile that in. The unrecognized struct value errors sounds like something unrelated to this linking issue.

Perhaps it might be possible to stub in those standard library functions with your own implementation?

juj picture juj  ·  14 Jun 2014
0

Interesting, I suspected as much, but had hoped dylibs might have llvm
bitcode in there.

I'll take a look to see how much of the Swift standard library source is
available to compile from, but I fear it won't be enough.

Appreciate the response, thanks!

On Saturday, June 14, 2014, juj [email protected] wrote:

Unfortunately it is not possible to link .dylib files to Emscripten. The
reason for that is that the .dylibs already contain native machine code for
x86/x64, and Emscripten cannot "go back" and get that to LLVM IR form
again. What one would have to do is implement the standard library for
Swift and compile that in. The unrecognized struct value errors sounds like
something unrelated to this linking issue.

Perhaps it might be possible to stub in those standard library functions
with your own implementation?


Reply to this email directly or view it on GitHub
https://github.com/kripken/emscripten/issues/2427#issuecomment-46086938.

geelen picture geelen  ·  14 Jun 2014
0

What if we implement our own standard lib, according to http://practicalswift.com/2014/06/14/the-swift-standard-library-list-of-built-in-functions there are just 74 functions

iongion picture iongion  ·  5 Oct 2014
0

If the println command was omitted, would this work? (it didn't for me, I still got the same result, but I'm very ignorant) Then one could create a library and just call it from js without it depending on anything.

Also, an update, the commands in the first post seem deprecated, here is what I tried:

Versions

$ xcrun swiftc --version
Swift version 1.0 (swift-600.0.45.3.2)
Target: x86_64-apple-darwin13.2.0

$ emcc --version
emcc (Emscripten GCC-like replacement) 1.25.0 ()
Copyright (C) 2014 the Emscripten authors (see AUTHORS.txt)
This is free and open source software under the MIT license.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

The file to compile:

$ cat f.swift
var message = 1

The IR

$ xcrun swiftc -emit-ir f.swift
; ModuleID = '-'
target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-apple-darwin13.2.0"

%Si = type <{ i64 }>

@_Tv1f7messageSi = global %Si zeroinitializer, align 8

define internal void @top_level_code() {
entry:
  store i64 1, i64* getelementptr inbounds (%Si* @_Tv1f7messageSi, i32 0, i32 0), align 8
  ret void
}

define i32 @main(i32 %argc, i8** %argv) {
entry:
  %0 = call i8* @_TFSsa6C_ARGCVSs5Int32()
  %1 = bitcast i8* %0 to i32*
  store i32 %argc, i32* %1
  %2 = call i8* @_TFSsa6C_ARGVGVSs20UnsafeMutablePointerGS_VSs4Int8__()
  %3 = bitcast i8* %2 to i8***
  store i8** %argv, i8*** %3
  call void @top_level_code()
  ret i32 0
}

declare i8* @_TFSsa6C_ARGCVSs5Int32()

declare i8* @_TFSsa6C_ARGVGVSs20UnsafeMutablePointerGS_VSs4Int8__()

!llvm.module.flags = !{!0, !1, !2, !5, !6, !7, !8}

!0 = metadata !{i32 2, metadata !"Dwarf Version", i32 3}
!1 = metadata !{i32 1, metadata !"Debug Info Version", i32 1}
!2 = metadata !{i32 6, metadata !"Linker Options", metadata !3}
!3 = metadata !{metadata !4}
!4 = metadata !{metadata !"-lswiftCore"}
!5 = metadata !{i32 1, metadata !"Objective-C Version", i32 2}
!6 = metadata !{i32 1, metadata !"Objective-C Image Info Version", i32 0}
!7 = metadata !{i32 1, metadata !"Objective-C Image Info Section", metadata !"__DATA, __objc_imageinfo, regular, no_dead_strip"}
!8 = metadata !{i32 4, metadata !"Objective-C Garbage Collection", i32 0}

The SIL (IDK if it's useful, but can't hurt):

$ xcrun swiftc -emit-sil f.swift
sil_stage canonical

import Builtin
import Swift
import SwiftShims

var message: Int

// top_level_code
sil private @top_level_code : $@thin () -> () {
bb0:
  %0 = global_addr #message : $*Int               // user: %3
  %1 = integer_literal $Builtin.Word, 1           // user: %2
  %2 = struct $Int (%1 : $Builtin.Word)           // user: %3
  store %2 to %0 : $*Int                          // id: %3
  %4 = tuple ()                                   // user: %5
  return %4 : $()                                 // id: %5
}

// Swift.Int._convertFromBuiltinIntegerLiteral (Swift.Int.Type)(Builtin.Int2048) -> Swift.Int
sil public_external [transparent] @_TFSi33_convertFromBuiltinIntegerLiteralfMSiFBi2048_Si : $@thin (Builtin.Int2048, @thin Int.Type) -> Int {
bb0(%0 : $Builtin.Int2048, %1 : $@thin Int.Type):
  %2 = builtin_function_ref "s_to_s_checked_trunc_Int2048_Word" : $@thin (Builtin.Int2048) -> (Builtin.Word, Builtin.Int1) // user: %3
  %3 = apply %2(%0) : $@thin (Builtin.Int2048) -> (Builtin.Word, Builtin.Int1) // user: %4
  %4 = tuple_extract %3 : $(Builtin.Word, Builtin.Int1), 0 // user: %5
  %5 = struct $Int (%4 : $Builtin.Word)           // user: %6
  return %5 : $Int                                // id: %6
}

Trying to compile with emcc:

$ xcrun swiftc -emit-bc f.swift -o f.bc
$ emcc f.bc
WARNING: Linking two modules of different data layouts: '/Users/josh/.emscripten_cache/libc.bc' is 'e-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-p:32:32:32-v128:32:128-n32-S128' whereas '/Users/josh/deleteme/swift-play/f.bc' is 'e-m:o-i64:64-f80:128-n8:16:32:64-S128'
WARNING: Linking two modules of different target triples: /Users/josh/.emscripten_cache/libc.bc' is 'asmjs-unknown-emscripten' whereas '/Users/josh/deleteme/swift-play/f.bc' is 'x86_64-apple-darwin13.2.0'
Unknown specifier in datalayout string
UNREACHABLE executed at /Users/clb/emscripten-fastcomp/lib/IR/DataLayout.cpp:300!
0  opt                      0x00000001086d04ae llvm::sys::PrintStackTrace(__sFILE*) + 46
1  opt                      0x00000001086d07bb PrintStackTraceSignalHandler(void*) + 27
2  opt                      0x00000001086d0b4c SignalHandler(int) + 412
3  libsystem_platform.dylib 0x00007fff8b0e35aa _sigtramp + 26
4  libsystem_platform.dylib 0x00007fff6492d380 _sigtramp + 3649347056
5  opt                      0x00000001086d07eb raise + 27
6  opt                      0x00000001086d08a2 abort + 18
7  opt                      0x000000010865a7a6 llvm::llvm_unreachable_internal(char const*, char const*, unsigned int) + 198
8  opt                      0x0000000108416b74 llvm::DataLayout::parseSpecifier(llvm::StringRef) + 2804
9  opt                      0x0000000108415c57 llvm::DataLayout::init(llvm::StringRef) + 471
10 opt                      0x000000010749b47e llvm::DataLayout::DataLayout(llvm::StringRef) + 158
11 opt                      0x0000000107482ba5 llvm::DataLayout::DataLayout(llvm::StringRef) + 37
12 opt                      0x000000010747943c main + 3756
13 libdyld.dylib            0x00007fff8a2865fd start + 1
Stack dump:
0.  Program arguments: /Users/josh/code/emsdk_portable/clang/e1.25.0_64bit/opt /var/folders/7g/mbft22555w3_2nqs_h1kbglw0000gn/T/tmp93OFuV/a.out.bc -strip-debug -internalize -internalize-public-api-list=main,malloc,free -globaldce -pnacl-abi-simplify-preopt -pnacl-abi-simplify-postopt -enable-emscripten-cxx-exceptions -o /var/folders/7g/mbft22555w3_2nqs_h1kbglw0000gn/T/tmp93OFuV/a.out.bc.opt.bc
Traceback (most recent call last):
  File "/Users/josh/code/emsdk_portable/emscripten/1.25.0/emcc", line 1224, in <module>
    shared.Building.llvm_opt(final, link_opts)
  File "/Users/josh/code/emsdk_portable/emscripten/1.25.0/tools/shared.py", line 1357, in llvm_opt
    assert os.path.exists(target), 'Failed to run llvm optimizations: ' + output
AssertionError: Failed to run llvm optimizations:
JoshCheek picture JoshCheek  ·  20 Oct 2014
0

If anyone can look at the above and give me an idea of what specifically it's missing, ie "it's trying to link stdlib.swift" and a link to somewhere that someone documents how they solved a similar problem, then I'll put a few hours into trying to get it. If there's any reasonable progress in that time, I'll report it here and probably keep at it.

Right now I'm just so ignorant that I don't even know what what the problem is or what to look for in terms of resolving it. But I have like 50 options I'm considering for my project, including things like Elm which I've already put a few days into playing with, so I need some guidance to not wander in circles for a months.

JoshCheek picture JoshCheek  ·  20 Oct 2014
0

The targets need to match. As the IR shows, the swift frontend emits

target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-apple-darwin13.2.0"

Emscripten uses its own triple (asmjs-unknown-emscripten). You might be able to force the frontend to use it? If not, you _might_ be able to just edit the LLVM IR file in text format to fix it (create one using emcc, to see the right values for those two lines). But, this is risky, as the triple affects stuff that is baked into the output in various places...

kripken picture kripken  ·  20 Oct 2014
0

So, I'm ignorant enough that I don't know how to interpret "the targets need to match" (I don't know what "target" means, or what a "triple" is -- kinda sounds like a fixed-length set of memory, like a "tuple" in a functional language). So it's unclear to me if it's something that I fucked up (e.g. did I compile it wrong, or with my own version of some binary, since emscripten uses its own set of binaries -- e.g. python2 was not in the PATH, and I couldn't find it in the emsdk_portable, so I just made a bash script named python2 that then invoked python and forwarded the args. Seemed reasonable since I had Python 2.7.5 installed, which the readme alleged was compatible https://github.com/kripken/emscripten/blob/06961a0ef6e3d8d92d5e36ff904262fefec62bec/tests/python/readme.md, and xcrun is some binary that I don't understand, which is just sort of generally available, but didn't come from emscripten) or is it some issue caused by Swift being overly zealous about an OSX environment, or is it some flag to find to tell it to chill the fuck out and forget about being execuatble and just compile as if some other program will load it and deal with all the integration stuff? I mean, you'll notice all I do in this script is declare a variable and assign it the value 1, so what is it trying to dynamically load? And does that have anything to do with the "targets" not matching?

I'll create the file using emcc and diff them and spend an hour poring over the differences, confused and trying and failing repeatedly, to figure out what I need to do, but I don't know how. The bc format si the only one I've found so far that emcc will even consider trying to compile.

The reality is that I have no particularly good model for figuring out how this stuff works. I once spent 8 hours getting a C program to compile and load a library I was interested in, that's about the extent of my experience. Code that operates at this level operates under a model that I am simply ignorant of, so while we're in this domain, it's reasonable to me like I've never programmed before.

I'm actually pretty good at guessing and cursing repeatedly trying and failing and guessing and cursing again, but I need some sort of model or context to iterate upon. And there's a sufficiently large amount of information available out there, that I need someone who's familiar with this domain to clue me in, and imply to me that this isn't a giant waste of my time (b/c, lets be honest, I could just suck it up and write my shit in JavaScript).

JoshCheek picture JoshCheek  ·  21 Oct 2014
0

A target is what clang/LLVM is emitting code for. When you build normally, on OSX your target triple is something like darwin-intel-apple or some other combo that contains the information that you are building for intel hardware, to run on a darwin kernel on an apple userspace. This affects codegen in various ways, and the LLVM IR is not portable because of it.

Emscripten has its own target, and emcc tells clang and LLVM to use it, -triple=asmjs-unknown-emscripten or something like that.

kripken picture kripken  ·  21 Oct 2014
0

Hmm. Guess I thought LLVM was the target.

Anyway, asmjs-unknown-emscripten looks correct:

$ ag asmjs-unknown-emscripten | wc -l
     115

-triple isn't the correct flag, apparently, and it tells me the wrong help file:

$ xcrun swiftc -Xllvm "-triple=asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc
swift (LLVM option parsing): Unknown command line argument '-triple=asmjs-unknown-emscripten'.  Try: 'swift (LLVM option parsing) -help'
swift (LLVM option parsing): Did you mean '-spiller=asmjs-unknown-emscripten'?

$ swift '(LLVM option parsing)' -help
fish: Unknown command 'swift'

$ xcrun swift '(LLVM option parsing)' -help
<unknown>:0: error: no such file or directory: '(LLVM option parsing)'

$ xcrun swift 'LLVM option parsing' -help
<unknown>:0: error: no such file or directory: 'LLVM option parsing'

$ xcrun swift 'option parsing' -help
<unknown>:0: error: no such file or directory: 'option parsing'

llvm isn't a binary, but tab-complete suggestions include llvm-gcc and llvm-g++, their help screens don't mention a triple, but do talk about a --target, but that doesn't work either. Which is strange, because it's listed in the help output, so it seems like the error should be something about using it wrong, rather than "Unknown command line argument", so maybe this isn't the binary that it invokes for llvm:

$ xcrun swiftc -Xllvm "--target=asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc
swift (LLVM option parsing): Unknown command line argument '--target=asmjs-unknown-emscripten'.  Try: 'swift (LLVM option parsing) -help'
swift (LLVM option parsing): Did you mean '-stats=asmjs-unknown-emscripten'?

# same output for each of these potential variations:
$ xcrun swiftc -Xllvm "--target=asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc
$ xcrun swiftc -Xllvm "--target asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc
$ xcrun swiftc -Xllvm "-target asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc
$ xcrun swiftc -Xllvm "-target=asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc
$ xcrun swiftc -Xllvm "--target" -Xllvm "asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc

# for that last one, it says "swift (LLVM option parsing): Did you mean '-stats'?"
# so lets verify that llvm-gcc is the right binary by seeing if `-stats` is one of its options:
$ llvm-gcc --help | grep stats

# ...nope, what the fuck is llvm?
# searching implies its binary is named "clang" for some reason,
# but no dice here either:
$ clang --help | grep stats

# man pages have any ideas?
$ man llvm-gcc
No manual entry for llvm-gcc

$ man llvm-g++
No manual entry for llvm-g++

$ man clang # this one works!

# After looking through this man page, I try -arch, which doesn't do shit
# was going to try setting MACOSX_DEPLOYMENT_TARGET env var
# but then I realized there's some way to search all man pages for "-stats"
# some stupid searching (man man) eventually I figure out:
$ man -K stats

# go through this list, and `gcc-4.8` shows up, so maybe llvm is just gcc? *shrug* lets try it
# it turns out to be fruitless
# at this point, I'm considering trying to find in the emscripten code where they invoke it
# figure I'll try googling a bit, find some docs for `llc`, which isn't a binary on my system, but does have a `-mtriple` flag, so lets try that:
$ xcrun swiftc -Xllvm "-mtriple=asmjs-unknown-emscripten" -emit-bc f.swift -o f.bc

# run the gamut of possible ways that I'm supposed to pass this thing
# (why can't I find a fucking example of how to do this?)

# eventually, one of the options suggests "-spiller" as a correction possibility
$ man -K spiller  # ...just some thing about crypto

At this point, documenting in code would add another hour, but many searches, fancy -k and -K uses of the man flag. A google search for llvm -spiller and -stats takes me back to that llc page, look through it a _second_ time, this time notice that they have two dashes before them, even though the suggestion when I got it wrong only has 1 dash. So, try the permutations of possible ways to pass an argument to -Xllvm with two dashes. None of them work.

Get pissed, internet search like the fourth time for an example of how to use -Xllvm flag, no results, cd to xcode and search for the string 'Xllvm', nothing. Try emsdk_portable, nothing.

Look one more time at llvm-gcc, llvm-g++, lldb, apropos llvm, clang, nothing.

Decide to go see if I can find where this happens in emsdk_portable, looks like the makefiles set a build_triple but never use it. Probably some makefile magic, and I don't feel like reading a book about makefiles just to find out how this gets persisted. Notice some of the files use LLVM_TARGET, I don't know Python, maybe that's an env var, so try compiling with that set env LLVM_TARGET=asmjs-unknown-emscripten xcrun swiftc -emit-bc f.swift -o f.bc, but it doesn't do anything (triple is still set to "x86_64-apple-darwin13.2.0"), remember the clang man pages had an env var that seemed like it had potential, so env MACOSX_DEPLOYMENT_TARGET=asmjs-unknown-emscripten xcrun swiftc -emit-bc f.swift -o f.bc. Nope.

So, I'm like 3 hours in, and literally no further than when I started. Looks like someone got Rust compiling to asm.js, maybe that's a better choice. Someone got Go compiling directly to JavaScript. Might be sufficient. Could go suffer through the Elm type system for another week or two (it was basically like this: hours of slamming myself into the wall, but I kept at it for like 3 times longer), or maybe write the thing in C, since that at least compiles, but some part of me thinks that's masochism. If I did it in Opal, I wouldn't have to learn a new language... but that's part of the appeal, and I'm not sure it'll be able to scale in directions I might want to go with my project.

Going to go for a drink and weigh the shittiness of C as a language against the fact that I'll almost certainly succeed.

JoshCheek picture JoshCheek  ·  22 Oct 2014
0

I was able to get very, very simple programs to build with the 6.3 beta at least. Unfortunately even something as simple as a string literal will not work, and regardless the resulting file has a bunch of missing symbols. Some of the symbols (swift_once) are defined in libswiftCore.dylib, so it's likely using Swift will be impossible until the standard library is open sourced.

endash picture endash  ·  21 Feb 2015
0

So, luckily, this will help later on: https://developer.apple.com/swift/blog/?id=29

guidobouman picture guidobouman  ·  13 Jul 2015
0

So I was trying to do the same thing, and I just stumbled upon this Issue. So glad that I won't waste my time again.

By the way did you try again with XCode 7 and the swiftc tool??

nmn picture nmn  ·  16 Jul 2015
0

Since Apple now supports LLVM Bitcode, would that enable including dylibs that contain it?

stepanhruda picture stepanhruda  ·  6 Oct 2015
0

LLVM bitcode is not portable. Bitcode for the App Store would be specific to Apple's hardware and OS.

kripken picture kripken  ·  6 Oct 2015
0

Thanks, that makes sense. In case anyone is interested in more details as I was, LLVM FAQ: Can I compile C or C++ code to platform-independent LLVM bitcode?

stepanhruda picture stepanhruda  ·  8 Oct 2015
0

It's been 18 months since I started this issue and I still really want this to happen. With Swift going open-source today let's hope that means it's now possible!

https://github.com/apple/swift :tada:

geelen picture geelen  ·  4 Dec 2015
0

This will definitely make it a lot easier! :)

Ok, to move forward here, we need to do the following:

  1. Figure out how to use emscripten's llvm with swift's llvm. Does swift have a fork of llvm, or can it use stock? If the latter, then this should be easy, just make swift use emscripten's llvm. We just need to see which version of llvm it expects (the incoming branch is on trunk llvm from last week, hopefully that is good and swift doesn't need an older tag).
  2. Get swift to emit code with the asmjs-unknown-emscripten triple.
  3. Get swift to stop at the bitcode stage.
  4. Run emcc on that bitcode and see what happens.
  5. We would need similar bitcode of whatever runtime libraries swift requires, except for libc etc. which we already have.
kripken picture kripken  ·  4 Dec 2015
0

Swift does have a fork of LLVM at apple/swift-llvm. Not sure what changes they have made.

Gaelan picture Gaelan  ·  4 Dec 2015
0

the swift-cli has a bunch of handy options to output just the llvm byte code etc.

nmn picture nmn  ·  4 Dec 2015
0

Let me know what I can help with would love to have something working!

jcampbell05 picture jcampbell05  ·  6 Dec 2015
0

Thanks for the info @Gaelan. Ok, if they have their own fork, then the best workflow would probably be:

  1. Upstream our target triple to LLVM
  2. When they merge in new LLVM, they'll get that
  3. Get Swift to emit bytecode to our target triple
  4. Run that bytecode through emcc

The first step is to upstream our triple, which is already partially there (due to nacl upstreamings), so it's a small patch and probably not controversial. What we need to send is basically this:

https://gist.github.com/kripken/0b7ba068faf21d5449a3

Anyone interested to help upstream that?

kripken picture kripken  ·  6 Dec 2015
0

It looks like Apple's llvm clone is pretty strict about changes being made upstream unless they are specifically swift-related. See: https://swift.org/contributing/#llvm-and-swift

ianyh picture ianyh  ·  7 Dec 2015
0

Could open an issue in the Swift LLVM github to ask them for changes

jcampbell05 picture jcampbell05  ·  7 Dec 2015
0

I think that reduces to asking them to merge the changes in themselves, which is against their policy. The patch would have to be submitted directly to llvm by someone here.

I have no familiarity with the llvm community, so I don't know how they would feel about, for example, introducing asmjs as an architecture. Anyone else have thoughts?

ianyh picture ianyh  ·  7 Dec 2015
0

This is very exciting @kripken! In theory, should we be able to make these modifications locally and try things out using the swift LLVM clone? I modified the relevant files in the swift LLVM clone, according to your patch - here is the diff of the changes I made: https://gist.github.com/sberan/43bc5fbff78ea47658f0

Unfortunately I don't see a new target for asmjs in the swift recompiled LLVM:

./build/Ninja-DebugAssert/llvm-macosx-x86_64/bin/llc --version
LLVM (http://llvm.org/):
  LLVM version 3.8.0svn
  DEBUG build with assertions.
  Built Dec  6 2015 (19:49:40).
  Default target: x86_64-apple-macosx10.9
  Host CPU: haswell

  Registered Targets:
    aarch64    - AArch64 (little endian)
    aarch64_be - AArch64 (big endian)
    arm        - ARM
    arm64      - ARM64 (little endian)
    armeb      - ARM (big endian)
    thumb      - Thumb
    thumbeb    - Thumb (big endian)
    x86        - 32-bit X86: Pentium-Pro and above
    x86-64     - 64-bit X86: EM64T and AMD64

Maybe more work needs to be done here in order to register asmjs as a target? Forgive me if this is a noob question, I'm very new to llvm and emscripten, just very excited to try swift in the browser :smile:

I'd be happy to try and push these changes upstream, but I'd like to verify that they'll work first! :laughing:

sberan picture sberan  ·  7 Dec 2015
0

I'll try to give it a go when I get a chance!

ianyh picture ianyh  ·  7 Dec 2015
0

I think you might not see a target because you can't build a full executable for js. But you should be able to build with -target asmjs-unknown-emscripten -emit-llvm and it should emit llvm for that specific target triple.

kripken picture kripken  ·  8 Dec 2015
0

There is another possible path here. Upstream LLVM has a WebAssembly triple now. It is almost identical to our asm.js triple. It _should_ be possible for swift to build to the wasm-wasm-wasm triple, then import that bitcode into emscripten which can compile it to asm.js.

(We would still need to make sure it's the same LLVM version, or close enough, to avoid e.g. debug info changes. But emscripten merges upstream every week or two now, so that should be easy, if someone tells me what is a good time to merge for Swift, i.e., when Swift merges.)

Note that the wasm triple is still experimental, so you need to build LLVM with it enabled, from the list of experimental targets.

kripken picture kripken  ·  6 Jan 2016
0

@kripken - This is an overview of the release process for Swift https://swift.org/blog/swift-2-2-release-process/

jcampbell05 picture jcampbell05  ·  6 Jan 2016
0

Swift uses llvm 3.8.0. They have their branch and not exactly sure their policy but once they finish swift 2.2, they'll probably merge their branch to upstream trunk.

3.8.0 already has wasm as kripken said, so I was able to build swift-llvm with wasm64 enabled and pass basic tests, but struggling to build swift with llvm-wasm and build swift stdlibs with with llvm-wasm + emscripten, main reason is that their build system is huge but not so flexible. (There're lots places that have hard-coded config like `if system==darwin then build for Mac, else build linux kind of thing) But I'm new for their build system so I may miss something.

I just wanted to make sure the process I'm doing is in the right course.

  • Build llvm with wasm enabled (already done)
  • Setup emscripten to use llvm-wasm enabled (not the one bundled with emscripten)
  • Build swift with llvm-wasm cross compile enabled, build swift stdlibs with llvm-wasm + emscripten
naan picture naan  ·  1 Feb 2016
0

@naan:

  1. Try the wasm32 target, not 64. JS does not work well with 64-bit, and so the asm.js target is 32-bit.
  2. I would actually not use the llvm-wasm backend. It is not ready yet, e.g. stack support is not yet complete, and it is not yet tested on large codebases - likely many bugs to get through before it is stable.
  3. Instead, code built with the wasm32 target should be almost identical to code built with the asm.js target. So in theory, you can take a wasm32 bitcode file, and let emscripten's asm.js backend compile it.
  4. Need to make sure while doing all of this that the files are built not just with the right triple, but the right headers. emcc does that automatically, by replacing normal system headers with emscripten's portable headers.
kripken picture kripken  ·  1 Feb 2016
0

@kripken Thank you for the followup.

So, #4 is sounds a bit challenging with their build system for me... I need to research more about the build system itself first, but it might be easier to use emcc toolchains to build swift-stdlibs if it automatically does header swap? I guess emscripten-fastcomp is still llvm 3.7.0 base, but once it gets synced with 3.8.0, I assume I can use that to build swift-stdlibs with actual asmjs target.

naan picture naan  ·  1 Feb 2016
0

If it helps, emscripten's incoming branch is already at 3.9.0 (svn).

kripken picture kripken  ·  1 Feb 2016
0

Any updates on this?

Danappelxx picture Danappelxx  ·  4 Mar 2016
0

+1
On Thu, Mar 3, 2016 at 6:51 PM Dan Appel [email protected] wrote:

Any updates on this?


Reply to this email directly or view it on GitHub
https://github.com/kripken/emscripten/issues/2427#issuecomment-192071228
.

Sent from my iPhone

gvkhna picture gvkhna  ·  4 Mar 2016
0

+1

joekim picture joekim  ·  4 Mar 2016
0

I'm also super interested to hear about progress here!

DorianListens picture DorianListens  ·  4 Mar 2016
6

@naan I was peripherally involved in the SwiftAndroid port and we had to do the same things there. Basically we change the sysroot to the one provided by the Android NDK - this makes the swift build tool look at the necessary headers and binaries from that point and not from the host machine's toolchain. I haven't used emscripten before in a real project, but I imagine it works in much the same way.

There are three open pull requests / discussions that will be instrumental in getting Swift working with Emscripten:

[RFC] Port to Android #1442 - check this out for changing the sysroot
Enable cross-compiling #1398 - this allows us to compile e.g. OSX --> Linux, not sure if it's needed but good to know that it's being worked on.
Flag to statically link Swift standard library #730 - the first issue that came up in this thread was that libSwiftCore is dynamically linked by default. Reliably building a statically linked version will be vital to continuing here.

I won't have time to work on this for at least another month or so, but I'm interested to keep track and answer any questions if I can in the meantime.

ephemer picture ephemer  ·  16 Mar 2016
4

@ephemer

Thank you for the info! That's really good to know. I have been in another project for a while and didn't have much time looking into this since last month. I hope I can make time for this soon.

naan picture naan  ·  16 Mar 2016
0

Flag to statically link Swift standard library #730 is resolved & waiting for review.

ramki1979 picture ramki1979  ·  30 Mar 2016
0

Hello All,
I tried using clang 3.9.0 to emit the llvm ir with -target wasm32-wasm32-wasm32, but end up getting warning for different data layout and target triple. The question is if llvm WebAssembly target is usable as is to generate llvm ir to be fed to emscripten ?

sparkagain picture sparkagain  ·  2 May 2016
1

The warning is expected, but should not cause problems in practice. The important things are the same in those two triples. (Except perhaps for the size of long doubles, is that important in Swift?)

kripken picture kripken  ·  2 May 2016
0

https://github.com/apple/swift/pull/1442 - Android port has been merged.

jcampbell05 picture jcampbell05  ·  2 May 2016
0

@kripken Thanks a lot. I do not think its important in swift.
On a second thought I think it would be great if emscripten can read in bc files (not generated with asm-unknown-emscripten) and emit bc files compatible with asm-unknown-emscripten. Can you please tell me is this currently supported or how difficult it would be to implement this. One challenge could be to undo the existing target assumptions of the input bc file.

sparkagain picture sparkagain  ·  3 May 2016
1

It's not possible in general, as LLVM IR is not portable. When you build for e.g. 64-bit ARM, you are baking in things that simply cannot be undone later.

kripken picture kripken  ·  3 May 2016
0

@kripken do you have a slack channel or something similar for Emscripten discussions? A lot of the bits I referenced above are falling into place and I'd be interested in figuring out what else is missing from the picture. Specifically I'm interested with how the c++ stdlib is currently dealt with as a dependency (to compare with how swift's stdlib might fit in), and to what extent the swift runtime might get in the way.

I see a lot of potential for swift + webassembly + WebGL via Emscripten!

ephemer picture ephemer  ·  3 May 2016
0

No slack, but there is an irc channel.

The C++ stdlib is basically just libc++ compiled in when needed. The only "magic" it requires is unwinding for C++ exceptions, which we have a few special functions for (using JS exceptions).

kripken picture kripken  ·  4 May 2016
0

@kripken Thanks for the link to the IRC channel.

The bit about 'compiled in' is what I don't really understand yet. Do users depend on a static libc++ that gets compiled into the LLVM IR before emscripten has anything to do with it? Or do they link against a dylib that is somehow later provided automatically by emscripten?

ephemer picture ephemer  ·  6 May 2016
0

The former. We take user input files, find out what system libs they need like libc and libc++, then link all those into one big bitcode file and optimize that. Doing so can really help with code size. Then we compile that into asm.js.

For special functionality that can't be done in bitcode (which turns into compiled asm.js), like stack unwinding, we call out into JS. So we link in necessary "JS libraries" as well, after the process described above.

Emscripten does have support for dylibs, so in theory one could use libc++ that way, however it's significantly less efficient in terms of code size and startup speed. However dylibs are still useful in other situations.

kripken picture kripken  ·  6 May 2016
10

@kripken thanks for the info.

So from what I can see, swiftc will have to be compiled with knowledge of the relevant triple (you mentioned that wasm32-wasm32-wasm32 is probably the most reasonable choice here, because it's already in upstream llvm), and referencing the correct headers, much like in https://github.com/apple/swift/pull/1442.

As in the android port, we'd need a swift stdlib (libswiftCore) compiled based on the emscripten sysroot. The issue I can see here is the swift build system currently expects to build and use complete (not intermediate llvm-ir) libraries. Someone smarter than me should be able to figure their way around that though. I know there is an option to produce llvm-ir _as well_, just not sure about producing only that.

For Linux, https://github.com/apple/swift/pull/1891 has been merged, the last of three pull requests to support compiling swift with -static-stdlib (libswiftCore). That said, I gather it was possible to emit just the llvm-ir already, so maybe this isn't a huge step forward after all.

On the Mac side (or maybe generally, I haven't looked into it yet) we do have https://github.com/apple/swift/pull/1398, which seems to enable cross-compilation based on an arbitrary sysroot and target. This is exactly what we need, maybe limited by the 'full compilation' issue mentioned above. See specifically https://github.com/apple/swift/pull/1398/commits/a4df003038b454d486bcb5fa8ef061e5ba446908 - I don't have time to look into this right now but maybe someone who does could give this process a go, putting in wasm32-wasm32 as the target and pointing to the emscripten sysroot at ...emsdk_portable/emscripten/1.35.0/system. We'd also need to point cross-compile-toolchain-bin to emscripten's clang. Just maybe we're close enough already...

ephemer picture ephemer  ·  8 May 2016
0

Any news? (:

paulofaria picture paulofaria  ·  8 Aug 2016
0

Does this help anything? https://hacks.mozilla.org/2016/10/webassembly-browser-preview/

@kripken @ephemer I'd love to push this forward if you could point me in the right direction!

paulofaria picture paulofaria  ·  1 Nov 2016
2

@paulofaria thanks for your enthusiasm. There's a fair bit of work being done on the swift build system at the moment to allow for cross compilation, that will bring us closer to building for webassembly.

It's a lot of work that involves changing a running system at its core but it'll get there

ephemer picture ephemer  ·  1 Nov 2016
0

any update?

goloveychuk picture goloveychuk  ·  13 Feb 2017
-1

@goloveychuk not that I'm aware of

ephemer picture ephemer  ·  16 Feb 2017
0

Any news ?

cacaodev picture cacaodev  ·  17 Jun 2017
1

I don't know how relevant this is for the question, but Swift Package Manager now supports cross compilation. I've built X toolchains for compiling Raspberry Pi Swift binaries on macOS and the reverse. Both however require the availability of Swift binaries (which we build on Raspi instead of x-compiling Swift itself).

My understanding is that CLang itself is always a cross compiler and comes w/ all supported targets builtin (i.e. the standard macOS clang can build for the Raspi armhf target out of the box, you just have to bring the toolchain). So if

Upstream LLVM has a WebAssembly triple now.

I suppose a lot may come for free. Has that 'upstream' be part of a release yet? Maybe the Xcode 9 one includes it? (is there a way to list the supported architectures by clang?)

Also, I think Swift 4 SPM defaults to building static libraries now.

helje5 picture helje5  ·  17 Jun 2017
1

Swift running in the browser at native-code speed? Heck yeah, let's make this happen!

I've downloaded the swift compiler and the emscripten sdk. I've successfully used emscripten to build and run "Hello world" in C. I tried passing "-target wasm32-wasm32-wasm32" to the latest Xcode 9 swiftc but it didn't work.

I'd like to do the process that @ephemer suggested above but I'll need a little guidance. Can you elaborate a bit on what the steps would be?

n8gray picture n8gray  ·  24 Jun 2017
10

Oh hey, wasm in WebKit: https://webkit.org/blog/7691/webassembly/

Also, the Xcode 9 build of Clang supports the wasm32 and wasm64 targets:

% xcrun clang --version
Apple LLVM version 9.0.0 (clang-900.0.26)
Target: x86_64-apple-darwin16.6.0
Thread model: posix
InstalledDir: /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

% xcrun clang -emit-llvm --target=wasm32 -Oz fib.c -c -o fib.bc
n8gray picture n8gray  ·  26 Jun 2017
0

...any updates?

Danappelxx picture Danappelxx  ·  9 Feb 2018
0

I tried to compile swift llvm ir code to wasm via emscripten - some syntax problems.
I guess we should try to build apple/swift-llvm and upstream llvm from source and try to compile ir code.
But this will likelly fail, sine apple frontend itself should support wasm32 target. This will require porting apple/swift for wasm32 target (same it was with android, but looks like, will requiremore significant changes)

goloveychuk picture goloveychuk  ·  9 Feb 2018
1

Achieved small success.
1) build llvm from sources. https://github.com/yurydelendik/wasmception
2) swiftc -emit-ir -Onone code.swift -o code.ll
3) open code.ll and remove

!5 = !{i32 6, !"Linker Options", !6}
!6 = !{!7, !8, !9}
!7 = !{!"-lswiftCore"}
!8 = !{!"-lswiftSwiftOnoneSupport"}
!9 = !{!"-lobjc"}

and !5 from

!llvm.module.flags = !{!0, !1, !2, !3, !4, !5, !10, !11}

(looks like swift-llvm is older than upstream llvm)
4) try llc -mtriple wasm32 code.ll
you will see WebAssembly doesn't support non-C calling conventions
So problem is that swift using swiftcc calling convention. Wasm supports only ccc calling convention.
https://llvm.org/docs/LangRef.html#calling-conventions
(if you comment this function llc will produce .s file.)
@kripken do you know, is it really possible to bring swiftcc convention to wasm target?

goloveychuk picture goloveychuk  ·  10 Feb 2018
0

The calling conventions are mostly for the backends, so it should be possible to just ignore it. That is, if swift didn't emit it, it should be ok, and for now you can strip it from the IR manually for testing. (There are exceptions like tail call optimizations that are done on IR and depend on the convention, but I don't think that would be an issue here.)

If you do that, is the output runnable? The simplest way to check is probably to use the wasm backend with emscripten instead of calling llc and s2wasm etc. yourself, that is, run emcc fixed.ll for LLVM IR with the calling convention stuff fixed up, and where emcc is set up to use latest LLVM built with the wasm backend (can set the LLVM env var to point it to that build, or edit ~/.emscripten).

kripken picture kripken  ·  10 Feb 2018
0

ok, i tried emcc.
After swiftc we have

define i32 @main(i32, i8**) #0 {
entry:
  %2 = bitcast i8** %1 to i8*
  ret i32 0
}

define hidden swiftcc void @_T04code4mainyyF() #0 {
entry:
  br label %0

; <label>:0:                                      ; preds = %entry
  br label %2

; <labe
...

So we have signature and implementation separately. Emcc catches first function, so we have no output. Ok, I merged it to one function.
This is whole code.

; ModuleID = 'code.ll'
source_filename = "code.ll"
target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
target triple = "asmjs-unknown-emscripten"

%struct._SwiftEmptyArrayStorage = type { %struct.HeapObject, %struct._SwiftArrayBodyStorage }
%struct.HeapObject = type { %struct.HeapMetadata*, %struct.InlineRefCounts }
%struct.HeapMetadata = type opaque
%struct.InlineRefCounts = type { i32, i32 }
%struct._SwiftArrayBodyStorage = type { i64, i64 }
%swift.type = type { i64 }
%swift.bridge = type opaque
%Any = type { [24 x i8], %swift.type* }
%TSi = type <{ i64 }>

@_swiftEmptyArrayStorage = external global %struct._SwiftEmptyArrayStorage, align 8
@_T0SiN = external global %swift.type, align 8
@__swift_reflection_version = linkonce_odr hidden constant i16 3
@llvm.used = appending global [1 x i8*] [i8* bitcast (i16* @__swift_reflection_version to i8*)], section "llvm.metadata", align 8

define i32 @main(i32, i8**) #0 {
entry:
  %2 = bitcast i8** %1 to i8*
  ; ret i32 0

  ; br label %0

; <label>:0:                                      ; preds = %entry
  ; br label %2

; <label>:1:                                      ; preds = %2
  br label %3

; <label>:2:                                      ; preds = %0
  ; br label %1

; <label>:3:                                      ; preds = %1
  %4 = call swiftcc { %swift.bridge*, i8* } @_T0s27_allocateUninitializedArraySayxG_BptBwlFyp_Tgq5(i64 1)
  %5 = extractvalue { %swift.bridge*, i8* } %4, 0
  %6 = extractvalue { %swift.bridge*, i8* } %4, 1
  %7 = call %swift.bridge* @swift_bridgeObjectRetain(%swift.bridge* %5) #2
  call void @swift_bridgeObjectRelease(%swift.bridge* %5) #2
  %8 = bitcast i8* %6 to %Any*
  %9 = getelementptr inbounds %Any, %Any* %8, i32 0, i32 1
  store %swift.type* @_T0SiN, %swift.type** %9, align 8
  %10 = getelementptr inbounds %Any, %Any* %8, i32 0, i32 0
  %11 = getelementptr inbounds %Any, %Any* %8, i32 0, i32 0
  %12 = bitcast [24 x i8]* %11 to %TSi*
  %._value = getelementptr inbounds %TSi, %TSi* %12, i32 0, i32 0
  store i64 578, i64* %._value, align 8
  %13 = call swiftcc { i64, i64, i64 } @_T0s5printySayypGd_SS9separatorSS10terminatortFfA0_()
  %14 = extractvalue { i64, i64, i64 } %13, 0
  %15 = extractvalue { i64, i64, i64 } %13, 1
  %16 = extractvalue { i64, i64, i64 } %13, 2
  %17 = call swiftcc { i64, i64, i64 } @_T0s5printySayypGd_SS9separatorSS10terminatortFfA1_()
  %18 = extractvalue { i64, i64, i64 } %17, 0
  %19 = extractvalue { i64, i64, i64 } %17, 1
  %20 = extractvalue { i64, i64, i64 } %17, 2
  call swiftcc void @_T0s5printySayypGd_SS9separatorSS10terminatortF(%swift.bridge* %5, i64 %14, i64 %15, i64 %16, i64 %18, i64 %19, i64 %20)
  ret i32 0
}

; Function Attrs: noinline
declare swiftcc void @_T0s5printySayypGd_SS9separatorSS10terminatortF(%swift.bridge*, i64, i64, i64, i64, i64, i64) #1

declare swiftcc { %swift.bridge*, i8* } @_T0s27_allocateUninitializedArraySayxG_BptBwlFyp_Tgq5(i64) #0

declare %swift.bridge* @swift_bridgeObjectRetain(%swift.bridge*)

declare void @swift_bridgeObjectRelease(%swift.bridge*)

; Function Attrs: noinline
declare swiftcc { i64, i64, i64 } @_T0s5printySayypGd_SS9separatorSS10terminatortFfA0_() #1

; Function Attrs: noinline
declare swiftcc { i64, i64, i64 } @_T0s5printySayypGd_SS9separatorSS10terminatortFfA1_() #1

attributes #0 = { "no-frame-pointer-elim"="true" "no-frame-pointer-elim-non-leaf" "target-cpu"="core2" "target-features"="+cx16,+fxsr,+mmx,+sse,+sse2,+sse3,+ssse3,+x87" }
attributes #1 = { noinline "no-frame-pointer-elim"="true" "no-frame-pointer-elim-non-leaf" "target-cpu"="core2" "target-features"="+cx16,+fxsr,+mmx,+sse,+sse2,+sse3,+ssse3,+x87" }
attributes #2 = { nounwind }

!llvm.module.flags = !{!0, !1, !2, !3, !4, !10, !11}

!0 = !{i32 1, !"Objective-C Version", i32 2}
!1 = !{i32 1, !"Objective-C Image Info Version", i32 0}
!2 = !{i32 1, !"Objective-C Image Info Section", !"__DATA, __objc_imageinfo, regular, no_dead_strip"}
!3 = !{i32 4, !"Objective-C Garbage Collection", i32 1280}
!4 = !{i32 1, !"Objective-C Class Properties", i32 64}
!10 = !{i32 1, !"PIC Level", i32 2}
!11 = !{i32 1, !"Swift Version", i32 5}

It compiles with emcc but writes messages like
warning: unresolved symbol: _T0s27_allocateUninitializedArraySayxG_BptBwlFyp_Tgq5
So, problem with linking. This could be caused by swiftcc or because I removed

!5 = !{i32 6, !"Linker Options", !6}
!6 = !{!7, !8}
!7 = !{!"-lswiftCore"}
!8 = !{!"-lswiftSwiftOnoneSupport"}
goloveychuk picture goloveychuk  ·  10 Feb 2018
0

Ok, yeah, that's a runtime linking issue then (the !5 = ! etc. things at the end are metadata, and can be ignored for now). Swift has a runtime that implements necessary things, like I guess _T0s27_allocateUninitializedArraySayxG_BptBwlFyp_Tgq5 allocates an uninitialized array, and there is an "Objective C bridge" of some sort.

Is there an open source version of the swift runtime library? If so then the next step is to build that (to bitcode, and link that to the main program). That runtime might need to be ported to wasm as well, but if it's already portable enough to run on linux and macos then it might just work, or might only take a little work.

kripken picture kripken  ·  10 Feb 2018
0

there is an "Objective C bridge" of some sort.

It may be better to not use the macOS compiler but do this on Linux. On macOS Swift has Objective-C integration, which is not there on Linux.

Is there an open source version of the swift runtime library?

At least the Linux stuff is all open source and available here: https://github.com/apple/swift

helje5 picture helje5  ·  11 Feb 2018
0

ok, let's forget for now about linking, calling conventions, std/runtime libs.
Here is simple swift code without using stdlib and cleaned llvm ir.
https://gist.github.com/goloveychuk/76f842045f28a19c939b210937dce2d4
It fails both with upstream llvm and built-in llvm

Built-in:

Not handled: i64 sub (i64 ptrtoint ([13 x i8]* @2 to i64), i64 ptrtoint (<{ i32, i32, i32, i32, i32, i32, i32, i32, i32, i32, i32 }>* @_T04code5StrctVMn to i64))
LLVM ERROR: FlattenGlobals: ConstantExpr opcode not handled: sub

Which is ok, since emscripten's mostyle targeting c/cpp code.

With upstream llvm:

DEBUG:root:emscript: binaryen s2wasm: /Users/badim/github/wasm/emsdk-portable/clang/e1.37.33_64bit/binaryen/bin/s2wasm /tmp/emscripten_temp/tmp0FUxXQ.wb.s --emscripten-glue --global-base=1024 --initial-memory=16777216 -l /Users/badim/.emscripten_cache/wasm/wasm_libc_rt.a -l /Users/badim/.emscripten_cache/wasm/wasm_compiler_rt.a --trap-mode=allow
[[object_alias:]]:
==========
.int32  2
    .int32  3
    .int32  (.L__unnamed_2-_T04code5StrctVMn)
==========

Included .s in gist

goloveychuk picture goloveychuk  ·  11 Feb 2018
0

Looks like that's a bug in s2wasm. It's actually being deprecated, though, so it's probably best to look at the new path, which is to use lld. Try building with EMCC_EXPERIMENTAL_USE_LLD=1 in the environment.

kripken picture kripken  ·  12 Feb 2018
2

Great! It compiled with several warnings

warning: unresolved symbol: _T0SiN
warning: unresolved symbol: _swift_slowAlloc
warning: unresolved symbol: _swift_slowDealloc

caused by linking problem.
It running and fails in runtime bacause can't find those functions. Next step - fix linking with stdlib/runtime.

goloveychuk picture goloveychuk  ·  12 Feb 2018
3

Great progress!

I have been doing trying to get everything to compile and linking.

Here is what I have so far (without success)
https://github.com/patcheng/swift/tree/swift-to-wasm

But it requires
https://github.com/patcheng/clang/tree/swift-to-wasm
and
https://github.com/patcheng/llvm/tree/swift-to-wasm

I had to build my own compile-rt, libcxx, libcxxabi and icu4c.

Here are the things I found (and attempted to address in my fork):

  • changed swift buildscripts to generate webassembly runtimes
  • change swift to have os(WebAssembly) and arch(wasm32) and arch(wasm64)
  • changed the runtime using those definitions.
  • clang and llvm doens't support swiftcall
  • llvm was not output wasm in a byte-aligned way that clang_ast expects.
  • llvm was not not generation atomic operations
  • llvm didn't have an asm parser yet.
  • llvm was failing to handle some "expression" that swift generates when compiling

Things are "compiling" and "linking".
Known issues:

  • The generated wasm file does not work. Firefox is compiling that the memory is not shared. It's probably missing in llvm/lld.
  • the generated wasm file is failing wasm-validation. The atomic operations are bad.
    probably a lot more things are broken.
patcheng picture patcheng  ·  13 Feb 2018
4

@patcheng nice work - sounds promising. It seems that swift-clang & swift-llvm are now up to date with release_60 which includes WASM changes. You should be able to have your branches forked from those instead.

I've rebased your changes on those here if you want to use it:

Note, I did have to resolve some conflicts, so you may want to verify. Also I didn't have what I needed to resolve conflicts in combine-sdiv.ll so you may need to resolve that yourself (I suspect it's mostly generated code).

It'd be great to get some of these changes merged onto https://github.com/apple 😄

therealbnut picture therealbnut  ·  10 Mar 2018
9

What is the status on this? Is there any way in which I can help? If so, where should I start?

benaubin picture benaubin  ·  13 May 2018
28

LLVM 7.0 was released yesterday with a lot of progress for WebAssembly target support. While the backend is still marked as experimental, not as much patching of upstream LLVM repositories should be needed. I've pulled 7.0 updates into Apple swift's fork repositories and rebased the changes in the main swift repository by @patcheng, while also fixing a few compilation issues.

Now the whole fresh toolchain with patches from above comments compiles for me, although I haven't checked yet if the actual WebAssembly support has improved. Going to look into that soon. I'm doing the work in these forks and branches:

https://github.com/maxdesiatov/swift
https://github.com/maxdesiatov/swift-compiler-rt
https://github.com/maxdesiatov/swift-llvm
https://github.com/maxdesiatov/swift-clang
https://github.com/maxdesiatov/lld

MaxDesiatov picture MaxDesiatov  ·  20 Sep 2018
0

Hey @maxdesiatov, would love to hear of any updates.

paulshapiro picture paulshapiro  ·  28 Oct 2018
17

Hey @paulshapiro, I appreciate your interest! I've got most if not all of Swift standard library compiling, as well as patches from @patcheng. I've also added support for Swift calling conventions when targeting WebAssembly in my clang and llvm forks. The main problem now is linking the standard library with its dependencies when compiled to WebAssembly. The build process gets a bit complicated though, as Swift stdlib also depends on libc (musl fork supporting WebAssembly in this case), libc++ and libc++abi and ICU all compiled and linked together for WebAssembly target from scratch. It also looks like all of this can only be linked with lld.

Unfortunately, a build system used within Swift compiler infrastructure is not very modular, it uses a mix of shell scripts, Python scripts and CMake. In addition to that, it doesn't look like cross-compiling the whole toolchain from scratch is supported well.

I'm currently getting changes to the build system polished so that wider audience could just clone my fork and run a reproducible build. This doesn't mean that Swift standard library will be compiled, linked and working for WebAssembly end-to-end at that point, but at least the build system issues would become reproducible and allow more people to contribute their fixes to these problems.

MaxDesiatov picture MaxDesiatov  ·  28 Oct 2018
11

TWIMC, even when we're able to compile a few simple Swift executables to WebAssembly, there still a ton of infrastructure work than needs to be done for all this to work smoothly: Swift Package Manager support, helpers for managing multiple cross-compilation toolchains, IDE support, better bridging to JavaScript etc.

In my opinion, one of the most pressing issues is that it's not easy to produce a self-contained statically linked binary written in Swift be it a WebAssembly binary or an ELF binary for Linux. This is going to be a quite serious problem until we have dynamic linking working well in WebAssembly, which is probably not going to happen soon.

If you'd like to be able to easily distribute a self-contained WebAssembly builds of your Swift code, please support work being done on this (in the increasing order of complexity):

  1. Upvote Jira issues and PRs related to this:
  2. SR-648 - SPM support for statically linked binaries
  3. SR-3819 - cross-platform ICU bug that prevents static linking
  4. SR-237 - build system improvement that splits the giant shell script build-script-impl that controls the build process of Swift compiler infrastructure
  5. apple/swift#19860 - fix for SR-3819
  6. Reproduce these issues locally and provide relevant info in corresponding PRs and bug reports.
  7. Highlight importance of these issues on Swift Forums and wider in the community. These issues seem obscure and irrelevant if you're only targeting iOS and other Apple's platforms exclusively, but it's important to solve them for better support of other platforms.
  8. Provide fixes/PRs for these issues.

Thanks!

MaxDesiatov picture MaxDesiatov  ·  28 Oct 2018
9

I think I finally got the build script modified so that musl, icu4c, libcxx and libcxxabi are pulled and built automatically with a few more cross-compilation flags set. Unfortunately, there are still build errors caused by missing directories, which need to be investigated.

At least, no more manual steps or additional custom scripts are required now. You can reproduce this WebAssembly build (and errors until those are fixed) with this short script:

mkdir swift-source && cd swift-source && \
  git clone https://github.com/maxdesiatov/swift.git && \
  ./swift/utils/update-checkout --clone --scheme wasm-next \
    --target-platform WebAssembly && \
  time ./swift/utils/build-script --release-debuginfo --skip-build-benchmarks \
    --webassembly --libicu

This will pull the correct forks with correct branch scheme configured.

To rerun the build after you've done any changes you can do this:

time ./swift/utils/build-script --release-debuginfo --skip-build-benchmarks \
  --webassembly --libicu

This will also print the total build time, not the check out time though, but check out is supposed to be done only once anyway.

Be prepared for long build times for cold builds: on 2018 15" MBP with top CPU and RAM config it takes around 40 minutes until failing. I anticipate the complete successful build to take much more than this as it needs to assemble both the host macOS and the cross-compiled WebAssembly toolchains.

Thank you all for following. I'll post updates here when I get any significant fixes pushed to my fork. PRs welcome!

MaxDesiatov picture MaxDesiatov  ·  30 Oct 2018
0

For future reference, this is the error message during Swift stdlib compilation that's currently blocking me:

FAILED: stdlib/public/core/webassembly/wasm32/Swift.o 
cd /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core && /usr/bin/python /Users/maxd/Documents/swift-source-wasm/swift-source/swift/utils/line-directive @/Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/h4pYG.txt -- /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./bin/swiftc -c -sdk -target wasm32-unknown-unknown-wasm -resource-dir /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift -O -g -D INTERNAL_CHECKS_ENABLED -D SWIFT_ENABLE_RUNTIME_FUNCTION_COUNTERS -I /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/webassembly/wasm32 -module-cache-path /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./module-cache -no-link-objc-runtime -Xfrontend -enable-resilience -Xfrontend -enable-sil-ownership -nostdimport -parse-stdlib -module-name Swift -Xfrontend -group-info-path -Xfrontend /Users/maxd/Documents/swift-source-wasm/swift-source/swift/stdlib/public/core/GroupInfo.json -swift-version 5 -warn-swift3-objc-inference-complete -Xfrontend -verify-syntax-tree -Xfrontend -enable-operator-designated-types -Xllvm -sil-inline-generics -Xllvm -sil-partial-specialization -Xcc -DswiftCore_EXPORTS -warn-implicit-overrides -module-link-name swiftCore -force-single-frontend-invocation -Xcc -D__SWIFT_CURRENT_DYLIB=swiftCore -parse-as-library -o /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/webassembly/wasm32/Swift.o @/Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/h4pYG.txt
<unknown>:0: error: no such file or directory: 'wasm32-unknown-unknown-wasm'
[1039/1331] Generating /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/webassembly/wasm32/Swift.swiftmodule
FAILED: lib/swift/webassembly/wasm32/Swift.swiftmodule lib/swift/webassembly/wasm32/Swift.swiftdoc lib/swift/webassembly/wasm32/Swift.swiftinterface 
cd /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core && /usr/local/Cellar/cmake/3.12.3/bin/cmake -E remove -f /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/webassembly/wasm32/Swift.swiftmodule && /usr/local/Cellar/cmake/3.12.3/bin/cmake -E remove -f /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/webassembly/wasm32/Swift.swiftdoc && /usr/local/Cellar/cmake/3.12.3/bin/cmake -E remove -f /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/webassembly/wasm32/Swift.swiftinterface && /usr/bin/python /Users/maxd/Documents/swift-source-wasm/swift-source/swift/utils/line-directive @/Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/h4pYG.txt -- /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./bin/swiftc -emit-module -o /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/webassembly/wasm32/Swift.swiftmodule -emit-parseable-module-interface -sdk -target wasm32-unknown-unknown-wasm -resource-dir /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift -O -g -D INTERNAL_CHECKS_ENABLED -D SWIFT_ENABLE_RUNTIME_FUNCTION_COUNTERS -I /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/webassembly/wasm32 -module-cache-path /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./module-cache -no-link-objc-runtime -Xfrontend -enable-resilience -Xfrontend -enable-sil-ownership -nostdimport -parse-stdlib -module-name Swift -Xfrontend -group-info-path -Xfrontend /Users/maxd/Documents/swift-source-wasm/swift-source/swift/stdlib/public/core/GroupInfo.json -swift-version 5 -warn-swift3-objc-inference-complete -Xfrontend -verify-syntax-tree -Xfrontend -enable-operator-designated-types -Xllvm -sil-inline-generics -Xllvm -sil-partial-specialization -Xcc -DswiftCore_EXPORTS -warn-implicit-overrides -module-link-name swiftCore -force-single-frontend-invocation -Xcc -D__SWIFT_CURRENT_DYLIB=swiftCore -parse-as-library @/Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/h4pYG.txt
<unknown>:0: error: no such file or directory: 'wasm32-unknown-unknown-wasm'
[1049/1331] Compiling /Users/maxd/Documents/swift-source-wasm/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/macosx/x86_64/Swift.o
ninja: build stopped: subcommand failed.
./swift/utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting
MaxDesiatov picture MaxDesiatov  ·  30 Oct 2018
0

error: no such file or directory: 'wasm32-unknown-unknown-wasm'

my ignorant suggestion:
-target=wasm32-unknown-unknown-wasm
or
-target:wasm32-unknown-unknown-wasm
instead of
-target wasm32-unknown-unknown-wasm
?

pannous picture pannous  ·  30 Oct 2018
0

It doesn't look like the argument is passed incorrectly, otherwise I'd expect line-directive or its proxied swiftc invocation to explicitly complain about the incorrect argument.

My current guess is that something's missing in build-script-impl or one of the numerous CMake files and their cross-compilation settings. Those seem to fail to create this directory within the main build directory tree. I plan to have a closer look at this line-directive script and possibly to debug it and swiftc to see where exactly the attempt to address that directory occurs and what's the absolute path of it. Then we'd need to retrace the build scripts to understand what's the best setting to tweak.

MaxDesiatov picture MaxDesiatov  ·  30 Oct 2018
2

nice progress!
according to swiftc -h:

  -sdk <sdk>              Compile against <sdk>

so, looks like <sdk> is missing.

looking at SwiftConfigureSDK.cmake, configure_sdk_windows defines:

    set(SWIFT_SDK_${prefix}_ARCH_${arch}_PATH "/")

but we are not setting the variable in configure_sdk_webassembly.

patcheng picture patcheng  ·  1 Nov 2018
1

Thank you @patcheng, that's a great shout!

I've fixed that missing directory problem and a few more header import errors following after that in my fork. Here's the error I'm currently stuck at:

[1049/1331] Compiling /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/webassembly/wasm32/Swift.o
FAILED: stdlib/public/core/webassembly/wasm32/Swift.o 
cd /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core && /usr/bin/python /Users/maxd/Documents/swift-source/swift/utils/line-directive @/Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/ce5Xt.txt -- /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./bin/swiftc -c -sdk / -target wasm32-unknown-unknown-wasm -resource-dir /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift -O -g -D INTERNAL_CHECKS_ENABLED -D SWIFT_ENABLE_RUNTIME_FUNCTION_COUNTERS -I /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/webassembly/wasm32 -module-cache-path /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./module-cache -no-link-objc-runtime -Xfrontend -enable-resilience -Xfrontend -enable-sil-ownership -nostdimport -parse-stdlib -module-name Swift -Xfrontend -group-info-path -Xfrontend /Users/maxd/Documents/swift-source/swift/stdlib/public/core/GroupInfo.json -swift-version 5 -warn-swift3-objc-inference-complete -Xfrontend -verify-syntax-tree -Xfrontend -enable-operator-designated-types -Xllvm -sil-inline-generics -Xllvm -sil-partial-specialization -Xcc -DswiftCore_EXPORTS -warn-implicit-overrides -module-link-name swiftCore -force-single-frontend-invocation -Xcc -D__SWIFT_CURRENT_DYLIB=swiftCore -parse-as-library -o /Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/webassembly/wasm32/Swift.o @/Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/core/ce5Xt.txt
<unknown>:0: error: file '/Users/maxd/Documents/swift-source/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/module-cache/15MFOT6M03ZBK/SwiftShims-1V5299MP7JS42.pcm' is not a valid precompiled module file
/Users/maxd/Documents/swift-source/swift/stdlib/public/core/ArrayBody.swift:18:8: error: no such module 'SwiftShims'
import SwiftShims

Looking at this precompiled module file we can see that it's compiled to WebAssembly:

% file module-cache/15MFOT6M03ZBK/SwiftShims-1V5299MP7JS42.pcm
module-cache/15MFOT6M03ZBK/SwiftShims-1V5299MP7JS42.pcm: WebAssembly (wasm) binary module version 0x1 (MVP)

Not sure if that's any good, a bit more googling leads to this clang doc on precompiled modules. There's also a mention of an unmerged patch for WebAssembly precompiled modules here and an unresolved clang bugreport probably related to this problem, both posted by @patcheng by the way 👍

Probably need to try and apply that patch or figure another way to make clang consume these precompiled WebAssembly modules. Or maybe these modules shouldn't be compiled to WebAssembly altogether? 🤔

MaxDesiatov picture MaxDesiatov  ·  1 Nov 2018
0

Hey @MaxDesiatov, a big thanks for all the work you've been doing on this!

I was wondering what the current status of compiling Swift to WASM is right now. I lack much experience in this area, but I'm very excited to potentially see support for this, so I would be happy to attempt contributing toward these efforts, given some guidance on what needs to be worked on.

Thanks!

aemino picture aemino  ·  22 Dec 2018
0

I’m pretty sure the precompiled modules should be readable to the cross-compile host (in your case Mac OS?) so shouldn’t be in web assembly format.

This is a common pain when trying to cross-compile generally (host vs target), and the last I looked the swift toolchain was doing weird things in this regard because it doesn’t follow CMake’s standard- Apple was having toolchain build performance issues internally using it in that way.

So to fix your issue it’s possible it’s an LLVM bug but it may also be an issue with the CMake scripts (or what is being passed to them by build-script-impl). I haven’t looked at the build system in a while.

I’m still tied up with some other projects but I do look here periodically and will try to share what limited experience I have with the build system.

ephemer picture ephemer  ·  28 Dec 2018
7

I just discovered that @ddunbar has been creating PRs to add WASM support to Swift:

https://github.com/apple/swift-clang/pull/235
https://github.com/apple/swift/pull/20684
https://github.com/apple/swift/pull/20687

patcheng picture patcheng  ·  3 Jan 2019
4

Just a few, very minimal initial ones... this bug has the status: https://bugs.swift.org/browse/SR-9307

ddunbar picture ddunbar  ·  4 Jan 2019
5

Hi all, sorry for the delayed reply. There are few more blocking things that I've discovered while trying to make stdlib and a few basic examples compile:

  1. The standard library is built under assumption of availability of multi-threading and atomics, which is not true for WASM and won't be for long time until all browsers support multi-threaded WASM (and multi-threaded WASM isn't even standardized yet AFAIR?). As was mentioned in the linked Swift Forums topic, the best option is probably to create stubs/shims for all files that do #include <atomic> or maybe find some libc++ implementation that provides those stubs for us?

  2. The LLVM code that Swift's compiler generates in IRGen also contains instructions for atomics. Those need to be omitted during code generation as they can't be lowered to WASM binary code due to the same reasons mentioned in 1.

  3. Some of the instructions generated by Swift's IRGen also specify memory offsets that can't be lowered to WASM. I didn't have enough time to figure out if that's a temporary limitation of LLVM WASM backend or has to be fixed in Swift's IRGen similar to 2.

Hope this helps.

MaxDesiatov picture MaxDesiatov  ·  12 Feb 2019
2

It looks like short-term it would be great if we could push apple/swift#20684 PR towards being merged. Reviewing its code, commenting on the review, upvoting the PR, pinging the authors/reviewers once a week or at least once a month and sharing to your friends so that they do the same would definitely help. And yes, the Apple's contribution guide for the Swift compiler says we can ping people once a week 😃

  • Ping the change. If it is urgent, provide reasons why it is important to get this change landed and ping it every couple of days. If it is not urgent, the common courtesy ping rate is one week. Remember that you’re asking for valuable time from other professional developers.

Here's also quick guide if anyone's interested in investigating the rest of the issues I mentioned above. The gist is that you need to compile a Swift toolchain that supports -target wasm32-unknown-unknown-wasm flag passed to the Swift compiler (which is what apple/swift#20684 is about). It's available in my wasm-next branch here https://github.com/MaxDesiatov/swift/tree/wasm-next, you can clone the dependencies after checking out that main forked repository this way: ./swift/utils/update-checkout --clone --scheme wasm-next.

Or you can try @ddunbar's branch from the PR, but you need to check if it pulls fresh LLVM/Clang when doing update-checkout from that branch.

I haven't compiled any of these branches for quite some time so that's why some rebasing on top of the latest code and resolving conflicts might be needed.

After you've got the toolchain compiling successfully you can try compiling MicroStdlib test:

% ./build/Ninja-DebugAssert+stdlib-ReleaseAssert/swift-macosx-x86_64/bin/swiftc \
-target wasm32-unknown-unknown-wasm -c -force-single-frontend-invocation \
-parse-as-library -parse-stdlib -module-name Swift -emit-module \
-emit-module-path Swift.swiftmodule -o Swift.ll \
swift/validation-test/stdlib/MicroStdlib/Inputs/Swift.swift

This will generate a Swift.ll file with LLVM instructions that we need to "lower" to WebAssembly binaries. You can even start with commenting out all of the code in swift/validation-test/stdlib/MicroStdlib/Inputs/Swift.swift and compiling an empty file. If that works, you can uncomment the most simple code line by line to proceed towards next issue.

If all goes well at the previous stage you can then compile Swift.ll file to WebAssembly with ./build/Ninja-DebugAssert+stdlib-ReleaseAssert/llvm-macosx-x86_64/bin/llc Swift.ll. That's where you might stumble upon problems with instructions not supported by the WebAssembly LLVM backend. It looks like most of the time it's Swift compiler generating instructions that don't make sense for WASM (atomics and probably address offsets, but the latter needs to be confirmed). You can also refer to LLVM guide for more details on these instructions.

Instructions that don't make sense should be fixed in IRGen code, otherwise we need to report problems with WebAssembly backend to upstream LLVM, but fixing those locally in forked LLVM repositories in the meantime is fine too. The earlier those problems are reported, fixed and upstreamed to the main LLVM repository (not Apple's), the higher is the chance they end up in the imminent LLVM 8.0 release and we won't need to maintain fixes in our forks anymore.

MaxDesiatov picture MaxDesiatov  ·  13 Feb 2019
0
  1. The standard library is built under assumption of availability of multi-threading and atomics, which is not true for WASM and won't be for long time until all browsers support multi-threaded WASM (and multi-threaded WASM isn't even standardized yet AFAIR?). As was mentioned in the linked Swift Forums topic, the best option is probably to create stubs/shims for all files that do #include <atomic> or maybe find some libc++ implementation that provides those stubs for us?

Maybe could be simulated using green threads ?

jcampbell05 picture jcampbell05  ·  13 Feb 2019
0

Maybe could be simulated using green threads ?

It's not that the standard library needs green threads or multi-threading to make it work, it's just that reference counting and the rest of the runtime were never written for platforms like WASM (or AVR for example) in mind, so it guards everything with atomics and synchronization assuming you always have to be safe when passing ref-counted objects around. On WASM that's not a problem as long as you don't have threads, we could even throw out all of the atomics and synchronization, the problem would be to keep that forked version maintained. That's why I'd prefer to have the shims compiled conditionally with upstream stdlib source code if/when we have shims available for atomics. And this is what people from the Swift core team recommended to do anyway:

It would be interesting to add a configuration flag to disable threading support in the "normal" runtime and standard library. Nobody's done that yet to my knowledge. For that specific issue, though, it is incorrect to avoid using in a single-threaded system; I'm not sure why that #if is there. is still meaningful for a single-threaded system for atomicity relative to signals and other preemption mechanisms.

It’s unfortunate libc++ doesn’t have a single-threaded atomic implementation, but having a stubbed-out version of atomic, mutex, and other synchronization primitives would be nice to minimize the amount of conditional code we need in the runtime itself.

MaxDesiatov picture MaxDesiatov  ·  13 Feb 2019
1

I left a comment on the forum, gist being that wasm has plans for atomics, but not any time soon, so stubbing <atomic> sounds like the most pragmatic choice.

I’d avoiding trying to fork the things to not use atomics in the stdlib/runtime because it’s much more likely to change than atomics, and you can write/maintain NOP functions much more easily than maintaining a fork of the standard library.

therealbnut picture therealbnut  ·  13 Feb 2019
-1

Thanks for all the hard work trying to get this to work - Swift is a fantastic language that I would love to see running in the browser (although the 5 megabyte bundle size will make me think twice).

alflennik picture alflennik  ·  20 Feb 2019
1

I spent the past week looking into this: it looks like LLVM/WebAssembly's support for custom data sections isn't mature yet, so Swift's runtime type reflection won't work.

I got to the same stage as @patcheng's and @MaxDesiatov's ports: specifically, I can compile a simple Swift function that doesn't use the stdlib and run it in Firefox.

swiftwasm_first

but I'm stuck here now as well.

I had to comment out a lot of stuff in LLVM to get asserts to stop firing when writing the type information sections to the wasm file.This results in invalid wasm files when I compile any Swift file with a class or struct declaration in it... including the stdlib.

While it's possible to disable runtime type information in Swift (with the -disable-reflection-metadata flag), this would break pretty much everything in the stdlib. (I remember from the Swift 2 era that even println("hello world") would crash without runtime type info)

I know @patcheng sent some patches to LLVM to try to resolve this, but I'm not sure what they do or whether more changes are needed.

I wonder how other language with runtime types support, such as Go or Rust's Any trait, encode their type information in WebAssembly.

zhuowei picture zhuowei  ·  13 Apr 2019
0

I wonder if using something like Binaryen instead of LLVM would allow us to write the metadata properly 🤔

I know that Binaryen IR doesn't even use SSA, this would probably require writing whole codegen pass for it, but maybe still worth it? Does anyone here have any experience with Binaryen and can provide their opinion on this?

MaxDesiatov picture MaxDesiatov  ·  13 Apr 2019
2

Hi zhuowei,

I work on the llvm backend for WebAssembly. We are certainly interested in
any bugs you have found related to custom data sections. Could you open a
bug https://bugs.llvm.org/ and perhaps attach an example of some bitcode
that causes the error/assertions?

cheers,
sam

On Sat, Apr 13, 2019 at 12:04 AM zhuowei notifications@github.com wrote:

I spent the past week looking into this: it looks like LLVM/WebAssembly's
support for custom data sections isn't mature yet, so Swift's runtime type
reflection won't work.

I got to the same stage
https://github.com/swiftwasm/swift/tree/swiftwasm as @patcheng
https://github.com/patcheng's and @MaxDesiatov
https://github.com/MaxDesiatov's ports: specifically, I can compile a
simple Swift function that doesn't use the stdlib and run it in Firefox.

[image: swiftwasm_first]
https://user-images.githubusercontent.com/704768/56073919-515f7f00-5d5f-11e9-9350-daa76c153a4f.png

but I'm stuck here now as well.

I had to comment out a lot of stuff
https://github.com/swiftwasm/swift-llvm/tree/stable-swiftwasm in LLVM
to get asserts to stop firing when writing the type information sections to
the wasm file.This results in invalid wasm files when I compile any Swift
file with a class or struct declaration in it... including the stdlib.

While it's possible to disable runtime type information in Swift (with the
-disable-reflection-metadata flag), this would break pretty much everything
in the stdlib. (I remember from the Swift 2 era that even println("hello
world") would crash without runtime type info)

I know @patcheng https://github.com/patcheng sent some patches to LLVM
https://reviews.llvm.org/people/revisions/16900/ to try to resolve
this, but I'm not sure what they do or whether more changes are needed.

I wonder how other language with runtime types support, such as Go or
Rust's Any trait, encode their type information in WebAssembly.


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/emscripten-core/emscripten/issues/2427#issuecomment-482783300,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAfe5fpT2lriWLn623BjOAw8EOEhmdRsks5vgYF8gaJpZM4CEPAq
.

sbc100 picture sbc100  ·  15 Apr 2019
4

@sbc100 I don't have a Bugzilla account on LLVM yet; I'll apply for one soon.

In the meantime, here are the issues I ran into, with links to patches/workarounds. I'm planning to send some of these upstream when possible

  • MCBinaryExpr in relocations won't work

Swift has a bunch of data relocations that are (symbol - constant value); this becomes a MCBinaryExpr, which causes asserts when writing object.

patcheng has a patch to check for these at https://reviews.llvm.org/D42564; I have a terrible kludge for fixing them at https://github.com/swiftwasm/llvm-project/commit/d0a183f5f728d671a453ebba0ecb8a3241566b4f and https://github.com/swiftwasm/llvm-project/commit/3dc1a433498656191d3a2b58c84be91b38ec82cf

This one can be worked around by not passing the equivalent of that flag when compiling Swift

  • Comdat with custom sections don't work

Comdat only works when custom sections aren't specified. Comdat in LLVM's Wasm object writer expects unique section name for each comdat symbol; The default section names are ".rodata.(symbol name)", which does work. If I specify a custom name, it doesn't: https://gist.github.com/zhuowei/597b77698634aac9b302d43b5c464838

Worked around by not using Comdat (since Swift disables comdats on Linux/ELF and macOS anyways)

  • No PC-Relative relocations on Wasm

This isn't an LLVM issue; it's just that Swift writes relative addresses in the metadata table to avoid runtime relocations (ie (target address - current address)); this requires a relocation type referencing two symbols, which doesn't exist in Wasm.

I asked a Swift developer on Twitter and it seems the Windows port has some workarounds for this that I could look into.

  • No -gmodules support for precompiled header/modules on Wasm

Patcheng has a patch for this: https://reviews.llvm.org/D42233 but it predated custom sections support, so it was decided to wait for custom sections to be included and switch the patch to use that.

I've addressed the comment and switched that patch to using custom sections: my version of that patch is https://github.com/swiftwasm/swift-llvm/commit/de2966bb5b0b95639b028c8414b0fa30fc60d1cb and https://github.com/swiftwasm/swift-llvm/commit/502e7536efe71996d920058a3026f304efed4971

  • WebAssembly doesn't have swift calling convention enabled yet

ddunbar added Swift calling convention support to llvm, but it isn't turned on yet (maybe because there are no tests?)

I enabled it at https://github.com/swiftwasm/llvm-project/commit/ea5605adc66575b6d88c585febc30121cb098b11, and I'm currently trying to port the ARM32 swift-return.ll/swifterror.ll/swiftself.ll test cases to WebAssembly so I can submit that patch.

Thank you so much for your work on the WebAssembly backend!

zhuowei picture zhuowei  ·  15 Apr 2019
1

@MaxDesiatov A quick update on my progress:

The Stdlib links. ...For certain values of "links": I basically completely broke the metadata support to get it to link. Attempting to run the resulting WebAssembly file gives me:

[email protected]:~/Documents/wasmtime$ target/release/wasmtime ~/swift-source/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/hello.wasm
failed to demangle superclass of _ContiguousArrayStorage from mangled name 's28__ContiguousArrayStorageBaseC'
error while processing main module /home/zhuowei/swift-source/build/Ninja-RelWithDebInfoAssert/swift-linux-x86_64/bin/hello.wasm: Instantiation error: Trap occurred while invoking start function: wasm trap at 0x7fbe9489677d

Yeah, this needs a lot more work before swift_getTypeByMangledName would run.

All my changes are pushed to the SwiftWasm repo, and I'll setup CI and document how to build it soon.

zhuowei picture zhuowei  ·  20 Apr 2019
0

@zhuowei - I'm curious to hear more about your goals for the port. You're using wasi, so is this mostly for non-browser use cases? Is running on the web not a goal? (And, is using an API like OpenGL not a goal?)

kripken picture kripken  ·  11 May 2019
0

@kripken Thank you so much for your comments!

I'm looking to target the browser, with serverside being a nice to have.

I mostly used WASI because it's new and defaults to using LLVM's wasi-ld for linking, instead of Emscripten's bitcode-based emcc linker. (I know Emscripten can use wasm-ld as well, but I think? the standard libraries are all compiled for emcc?) I actually started with a Emscripten sysroot, and it didn't take too many changes to switch to WASI. Thus, I'm sure most of the work done for this port would also be applicable to an Emscripten-based port.

I was planning to modify WASI's browser polyfill for better interop with web APIs and JavaScript (since Emscripten's EM_ASM macros probably won't work in Swift anyways). I understand that Emscripten's the more mature choice right now for interoperating with the Web. I would really appreciate your expertise in what approach I should take to turn Swift into a language for the web.

zhuowei picture zhuowei  ·  11 May 2019
0

Interesting, thanks @zhuowei!

Yes, emscripten does support wasm-ld and the LLVM wasm backend - basically if you just point emscripten to a plain upstream LLVM build, it will use all that (on linux, you can already try it out using emsdk install latest-upstream). Then it uses wasm object files, links them with wasm-ld, etc. We will switch to that by default soon, actually!

Anyhow, the reason I asked is I'm curious how emscripten can help here. Sounds like using the LLVM wasm backend is a good step we're close to completing. Otherwise, I guess it depends on the applications you want to port. My general thoughts on the topic, for languages using LLVM:

  • If the goal is to do mostly just minimal pure computation on the web, then you don't need wasi or emscripten - you can just use LLVM (and manual wasm imports/exports for JS integration). I assume Swift needs non-trivial standard library support, though, so probably this one isn't relevant.
  • If the goal is server-side code in pure wasm, then wasi is probably best. Or, if you want wasm+JS and/or to use node.js, then emscripten would be better.
  • If the goal is code that uses some low-level libc APIs, then wasi may be good - since that's pretty close to the server-side case anyhow. For code size on the web, though, it will be smaller to use emscripten instead of wasi + the wasi web polyfill. I wonder if emscripten can do something like "port" wasi applications and optimize them? But I'm not sure that can be as good as just recompiling them from source with emscripten. So supporting both wasi and emscripten may be optimal.
  • If the goal is to run on the web and to have natural integration with web APIs, then you probably want something language-specific like emscripten's WebIDL binder or embind for C++, or Rust's wasm-bindgen. (Btw, I hope Swift can do a lot better than any of those, because Swift's memory management is in principle more compatible with the web, being much more automatic.)
  • If the goal is to run on the web and use native C APIs like OpenGL, pthreads, SDL, etc., then emscripten is the best option, as we have support for those in very web-specific ways. For example, emscripten's pthreads use workers, which is necessary on the web (I'd expect wasi's eventual pthreads support to focus more on "normal" native pthreads support, the way it works on servers, which is very different); also emscripten has an SDL2 port, optimized WebGL bindings for OpenGL, etc. etc.

Since you say your main goal is the browser, then maybe emscripten has a role to play here - let me know what you think. We'd definitely like to help here if it's relevant!

Btw, a question about your Swift PR: I see it has ifdefs on both __wasi__ and __wasm__. The wasm ones would work with both wasi and emscripten (and any other target that uses the LLVM wasm backend), so using the wasi one only where absolutely necessary could make it easier to use emscripten or other things later - maybe that's already how it's written?

kripken picture kripken  ·  12 May 2019
0

@kripken I did see the progress on the Wasm-ld stuff: I honestly was just trying to be hipster by using the latest shiny SDK...

Minimal pure computation was possible even before we started this port (see my comment above) - most of the work is mainly about getting the full stdlib working.

I originally started by targetting Emscripten, and switching to WASI was pretty much a find-and-replace (since both used Musl), so adding support for Emscripten is definitely something that we'll look into once we get the basic support upstreamed.

Yes, that's what I tried to do for defines: I tried to use the __wasm__ define for changes common to the wasm architecture or the wasm file format, and use __wasi__ for changes specific to the WASI sysroot/musl port.

I named the platform as just "Wasm" in the Swift stdlib, so the conditional compiles there are os(Wasm); I'll probably change it to os(Wasi) there as well to match.

I definitely agree that support for Emscripten would be great for Swift - we're definitely interested in working together to make that possible in the future.

Thank you so much for the comments and for all your work on Emscripten.

zhuowei picture zhuowei  ·  12 May 2019
0

@kripken @zhuowei Hi I’m really excited for these developments, but I’ve got a lot of questions. 🎉

I don’t know too much about Emscripten, but I read that Rust chose not to use it because of its weighty runtime (https://github.com/rust-lang/rust/pull/45905).

Do you mind giving a high level overview (or linking) what functionality the runtime provides?

Also, how can we minimize the weight of that runtime? Can we use dead code stripping, LTO, import just what we need from Swift?

Thanks for your insights ♥️

therealbnut picture therealbnut  ·  12 May 2019
3

@zhuowei Great, let me know when and how we can help :)

@therealbnut Sure, let me try to explain what the extra runtime stuff does here. Maybe a step back first though - note that Rust didn't remove Emscripten support, they added a second option. But yes, that new option is definitely where most of the activity is, as you said. This makes sense because of the different use cases:

  • Emscripten's main use case is existing code in C/C++ that uses native APIs like POSIX, OpenGL, SDL, etc.
  • Rust's main use case is new code written for the web.

This is simply because Rust is a new language - there aren't many large important native applications written in Rust yet. Most relevant applications are currently written in C++ - for example, the Unity and Unreal game engines.

The runtime that emscripten provides is exactly to support POSIX, OpenGL, SDL, etc. It consists of ports of those libraries, of new C APIs to make porting them easy, and of basic building blocks to enable all of that. All of this is very web-specific and includes a lot of complexity because of that - because it's not easy to implement pthreads using Workers or normal OpenGL using WebGL, etc. The web platform is weird!

Note that we already do LTO and other things to reduce code size. We even do dead code elimination across both the wasm and the JS ("metadce"). We also make sure to keep our JS code compatible with closure advanced optimizations for the best JS minification on that side, and use binaryen to minify the wasm. But we can do more, and we do have some unnecessary code in the runtime we'd like to remove (the ongoing MINIMAL_RUNTIME project - help accelerating that is welcome!). But in the end, if an application uses POSIX files for example, then there is just a bunch of code that is unavoidable. But it's definitely worth it if you have a large codebase you want to just work on the web - since the runtime is tiny compared to your own code anyhow, and it saves you porting your code to new APIs. Using the same APIs for native and web also means your code has fewer web-specific differences, one useful consequences of which is you can often debug issues you see on the web version in a native build.

That's how I see the extra value that emscripten brings here. It's incredibly important for C/C++ because existing apps need that, but maybe not for Rust since it's new code anyhow. But with that said, if Rust becomes important in native application development - which seems plausible - then maybe the next 1 million line game engine will be written in Rust, and then emscripten would be the right tool to port it. I don't know enough about the Swift ecosystem to know how it fits in all this, but as discussed above, it should be pretty easy to support both wasi and emscripten, leaving options open for both new code as well as porting native applications.

Btw, emscripten brings another, separate type of value as well - it's mostly the same team of developers that work on emscripten, the LLVM wasm backend, wasm-ld, and binaryen. Since it's the same people, I think it's natural that we do a good job of making emscripten use those other tools in an optimal way. In particular, emscripten makes sure to call wasm-ld and binaryen with carefully-chosen flags for optimizing for code size and speed, uses features like metadce that was mentioned before, etc. So the out-of-the-box code quality - just grab the emsdk and do emcc -Os - is very good with emscripten, while otherwise you'd need to do more toolchain tinkering yourself.

kripken picture kripken  ·  12 May 2019
0

@kripken Awesome, thank you very much for taking the time to respond, that answers a lot of my questions, and some I hadn’t thought of yet :)

therealbnut picture therealbnut  ·  13 May 2019
0

This issue has been automatically marked as stale because there has been no activity in the past year. It will be closed automatically if no further activity occurs in the next 7 days. Feel free to re-open at any time if this issue is still relevant.

stale[bot] picture stale[bot]  ·  12 May 2020