When I've started UT engine & editor about 1.5 years ago, I needed to decide about attaching editor to engine. I've decided to create editor in C# connected to game engine (which does all rendering in editor's subwindow) via sockets (see this [PL]). Now I know it's probably not the best choice.
Let's call it option 1. Benefits: WYSIWYP, no editing/GUI in main engine. But another problem arises. We need several "edit/debug" features anyway, like displaying objects selection, moving objects with mouse, click-to-select, visualise lights/markers/scripts. Game engine doesn't need to, for example, save models but still there is a lot of code unnecessary for normal gameplay and a lot of optimizations missed (for example you can't assume that static objects will remain static as level designer can move them).
There are of course other options. We could code editor as completly standalone application with its own renderer. It lets engine rendering code be super-well optimized but lacks very important feature: WYSIWYP. I believe it's no use as of 2010.
And, of course, you can code everything inside your engine (including level editor and other stuff). Well, I was somehow scared to do it as my C++ was way worse 2 years ago -- now I don't see this option as total madness :)
What would I do if I've started coding this toolset now? I would go 1+2. I would code editor with its own, quite simple renderer and connect with main engine via sockets only for "Preview" tab/window (or just let engine window run on other display). This option is also possible if you have engine running on PS3/X2Π and editor toolset on PC. Still some optimizations are gone (like non-static static objects) but many other are saved.
It's a little too late for me to start coding editor from scratch (maybe in version 2.0) but I have a little different solution: I have separate Object Editor (able to import models/skeletons/animations/physics and do some basic editing and setting up) which -- to be honest-but-not-so-modest -- is itself probably more advanced than many amateur engines. ;)
