First things first

In order to rewrite the VM of Net Core, I have to get the VM source code to compile under Clang. Then, using Piggy, I can start to convert the C++ code to C#. The first block of code I plan to work on is the PE file reader.

While I can build Coreclr on Windows through a VS Developer Command Window, Cmake fails to generate build files within a Mingw Bash shell. In fact, I had no idea whether Clang was used even on Linux, because build.sh is opaque: there are no options to the script to display compilations (at least not that I can see). Years ago, make had such an option, but it seems to have changed. It is now done via “make VERBOSE=1”.

But, using the Ubuntu WSL c:/Windows/System32/bash.exe, I was able to get the build working, and examine the compile commands. Note it’s important to install the various prerequisites, otherwise Coreclr won’t build.

The question is very simple: what compiler options do I need for Clang++ in order to compile VM/*.cpp using Piggy? After a bit of work, a typical compile is this:

pushd /mnt/c/Users/Kenne/Documents/coreclr/bin/obj/Linux.x64.Debug/src/vm/wks; /usr/bin/clang++ -DAMD64 -DBIT64=1 -DBUILDENV_CHECKED=1 -DDBG_TARGET_64BIT=1 -DDBG_TARGET_AMD64=1 -DDBG_TARGET_AMD64_UNIX -DDBG_TARGET_WIN64=1 -DDEBUG -DDEBUGGING_SUPPORTED -DDISABLE_CONTRACTS -DFEATURE_ARRAYSTUB_AS_IL -DFEATURE_CODE_VERSIONING -DFEATURE_COLLECTIBLE_TYPES -DFEATURE_CORECLR -DFEATURE_CORESYSTEM -DFEATURE_CORRUPTING_EXCEPTIONS -DFEATURE_DBGIPC_TRANSPORT_DI -DFEATURE_DBGIPC_TRANSPORT_VM -DFEATURE_DEFAULT_INTERFACES=1 -DFEATURE_EVENTSOURCE_XPLAT=1 -DFEATURE_EVENT_TRACE=1 -DFEATURE_HIJACK -DFEATURE_ICASTABLE -DFEATURE_ISYM_READER -DFEATURE_JUMPSTAMP -DFEATURE_LEGACYNETCF_DBG_HOST_CONTROL -DFEATURE_MANAGED_ETW -DFEATURE_MANAGED_ETW_CHANNELS -DFEATURE_MANUALLY_MANAGED_CARD_BUNDLES -DFEATURE_MULTICASTSTUB_AS_IL -DFEATURE_MULTICOREJIT -DFEATURE_MULTIREG_RETURN -DFEATURE_PAL -DFEATURE_PAL_ANSI -DFEATURE_PAL_SXS -DFEATURE_PERFMAP -DFEATURE_PERFTRACING=1 -DFEATURE_PREJIT -DFEATURE_READYTORUN -DFEATURE_REJIT -DFEATURE_STANDALONE_GC -DFEATURE_STANDALONE_SN -DFEATURE_STRONGNAME_DELAY_SIGNING_ALLOWED -DFEATURE_STRONGNAME_MIGRATION -DFEATURE_STUBS_AS_IL -DFEATURE_SVR_GC -DFEATURE_SYMDIFF -DFEATURE_TIERED_COMPILATION -DFEATURE_USE_ASM_GC_WRITE_BARRIERS -DFEATURE_USE_SOFTWARE_WRITE_WATCH_FOR_GC_HEAP -DFEATURE_WINDOWSPHONE -DFEATURE_WINMD_RESILIENT -DLINUX64 -DPLATFORM_UNIX=1 -DPROFILING_SUPPORTED -DUNICODE -DUNIX_AMD64_ABI -DUNIX_AMD64_ABI_ITF -DURTBLDENV_FRIENDLY=Checked -DWRITE_BARRIER_CHECK=1 -D_AMD64_ -D_BLD_CLR -D_DBG -D_DEBUG -D_SECURE_SCL=0 -D_TARGET_64BIT_=1 -D_TARGET_AMD64_=1 -D_UNICODE -D_WIN64 -I/mnt/c/Users/Kenne/Documents/coreclr/bin/obj/Linux.x64.Debug/src/vm/wks -I/mnt/c/Users/Kenne/Documents/coreclr/src/vm/wks -I/mnt/c/Users/Kenne/Documents/coreclr/src/vm -I/mnt/c/Users/Kenne/Documents/coreclr/src/pal/prebuilt/inc -I/mnt/c/Users/Kenne/Documents/coreclr/bin/obj -I/mnt/c/Users/Kenne/Documents/coreclr/src/pal/inc -I/mnt/c/Users/Kenne/Documents/coreclr/src/pal/inc/rt -I/mnt/c/Users/Kenne/Documents/coreclr/src/pal/src/safecrt -I/mnt/c/Users/Kenne/Documents/coreclr/src/inc -I/mnt/c/Users/Kenne/Documents/coreclr/src/strongname/inc -I/mnt/c/Users/Kenne/Documents/coreclr/src/inc/winrt -I/mnt/c/Users/Kenne/Documents/coreclr/src/debug/inc -I/mnt/c/Users/Kenne/Documents/coreclr/src/debug/inc/amd64 -I/mnt/c/Users/Kenne/Documents/coreclr/src/debug/inc/dump -I/mnt/c/Users/Kenne/Documents/coreclr/src/md/inc -I/mnt/c/Users/Kenne/Documents/coreclr/src/classlibnative/bcltype -I/mnt/c/Users/Kenne/Documents/coreclr/src/classlibnative/cryptography -I/mnt/c/Users/Kenne/Documents/coreclr/src/classlibnative/inc -I/mnt/c/Users/Kenne/Documents/coreclr/bin/obj/Linux.x64.Debug/src/inc -I/mnt/c/Users/Kenne/Documents/coreclr/src/pal/inc/rt/cpp -I/mnt/c/Users/Kenne/Documents/coreclr/src/nativeresources -I/mnt/c/Users/Kenne/Documents/coreclr/src/vm/amd64 -Wall -Wno-null-conversion -std=c++11 -g -O0 -fno-omit-frame-pointer -fms-extensions -fwrapv -fstack-protector-strong -ferror-limit=4096 -Werror -Wno-unused-private-field -Wno-unused-variable -Wno-microsoft -Wno-tautological-compare -Wno-constant-logical-operand -Wno-pragma-pack -Wno-unknown-warning-option -Wno-invalid-offsetof -Wno-incompatible-ms-struct -fsigned-char -nostdinc -fPIC -o CMakeFiles/cee_wks.dir/__/peimage.cpp.o -c /mnt/c/Users/Kenne/Documents/coreclr/src/vm/peimage.cpp; popd

This long, rambling command contains a wealth of information necessary to use Clang. With this in hand, I can now try to use Piggy.

Unfortunately, Clang of the code on Windows doesn’t work well. It seems easier to get Piggy to work under Linux than for Coreclr to compile with Clang on Windows.

Note, there is going to be a lot of work needed to develop the patterns required to do the conversion. Piggy itself will need to be extended to allow both Listener- and Visitor-patterns because the Visitor-pattern is insufficient.

Ken

A few notes on Coreclr

I’m starting to go through the details of Coreclr, first with System.Private.CoreLib. Here are a few notes…

  1. I do not understand how people in Microsoft can actually modify the Coreclr without using an IDE, but it’s not apparent whether they do or not. There are .sln files for Visual Studio 2017, but I can’t open any of them after a “git clone” for two reasons: (a) missing generated build files and code; (b) unset environmental variables. For …/coreclr/bin/obj/Windows_NT.x64.Debug/CoreCLR.sln, I can open the solution after doing a build. For …\coreclr\src\System.Private.CoreLib\System.Private.CoreLib.sln, I am able to hack my way past this problem after setting an environmental variable. Steps: (a) run “./build.cmd” from within a “Developer Command Prompt for VS 2017” cmd.exe, at the top-level directory of Coreclr after checking out the source from the Github.com repository. (b) set the environmental variable PYTHON to the Python source (try “command -v python”). With these changes, I can open and start using an IDE to investigate the code. It is completely impractical to navigate quickly around the source with a straight editor. Instructions should be somewhere e.g. in the same directory as the .sln file, or even in the .sln file. Bogus, but there you have it.
  2. The attribute [MethodImplAttribute(MethodImplOptions.InternalCall)] is used in 556 places, some nested in PLATFORM_WINDOWS. For Campy, most InternalCall’s should be rewritten in C# code, and what can’t be rewritten, available in a DLL for each platform. For example, for an Array create, we have “[MethodImplAttribute(MethodImplOptions.InternalCall)]
    private static extern unsafe Array InternalCreate(void* elementType, int rank, int* pLengths, int* pLowerBounds); ” Some form of array create must be offered by the runtime, and work for all the different platforms. The plan is to rewrite the entirety, the VM, type system, GC, etc. in C# code. There are 19 InternalCall methods for the Array class, all of which need to be rewritten.
  3. The method “[MethodImplAttribute(MethodImplOptions.InternalCall)] private static extern unsafe void InternalSetValue(void* target, object value);” can be easily rewritten and should be to unsafe C# code, which I wrote initially for Campy, but changed it after integrating the DNA runtime. This is just one example of how this code is not platform independent! Other examples are Array.Length, LongLength, and GetLength(int).
  4. A few weeks ago, I decided to check whether Net Core and Net Framework actually conform to Net Standard–as I do not believe anyone or anything and afraid of zilch (I ice climb for relaxation). It took a little while, but I am mostly convinced (http://codinggorilla.com/?p=1578). And, I now know what I have to do to get Campy to conform to Net Standard.
  5. Whatever changes I make to Coreclr, I’ll want to easily pick up all the latest changes. So, I’ll be cloning the repo and working with that.
  6. The guts of Net Core is the VM and type system itself, which will need to be entirely rewritten because, yes, it is not platform independent code (it doesn’t work on a GPU for example). The VM is described in the Net Core documentation. The VM and type system source is here.
  7. I was finally able to get a change to Array.Copy into a local System.Private.CoreLib.dll and run it. The instructions MS provides are pretty much junk–they don’t work. Debugging of CoreRun.exe is given here, but it’s pretty convoluted. Just open VS2017, then File|Open Project/Solution, then navigate and select the CoreRun.exe executable. Click on Properties, and fill in the args, which will be the Net Core console app you want to run. The go. Note, the steps to create a console app that tests your Net Core changes follows:
    1. Create a Net Core test program using the normal Dotnet.exe.
    2. Build the test program, and “publish” it with win-x64 self-contained executable. That should be in <published directory>.
    3. Build a local Coreclr with your modifications in System.Private.CoreLib.
      1. .\build.cmd
    4. cp …\coreclr\bin\Product\Windows_NT.x64.Debug\System.Private.CoreLib.dll <published directory>
    5. Run CoreRun.exe <published directory>\application .exe.
  8. I’ve noticed that in stepping through CoreRun.exe, F11 doesn’t work sometimes in stepping into a function, like CoreHost2::ExecuteAssembly(). You might need to step at the assembly level to get to C++ functions or set up your breakpoints beforehand.
  9. Let the fun begin! For starters, I’ll continue to make changes to Array and other types that use InternalCall functions, and see if I can make a local Net Core that works. Independently, I’ll also start to translate all C++ code in the VM using several automated translation tools to C# code and check what platform issues there are there. I have a lot of tools at my disposal, including Piggy, a powerful transformational system which I wrote, and which I will be extending to deal with automatic translations of C++.

–Ken